diff options
46 files changed, 3553 insertions, 3224 deletions
@@ -4,3 +4,4 @@ !/lib/ManUtils/ManUtils.xs indexer/target web/target +util/.config @@ -1,20 +1,661 @@ -Copyright (c) 2012 Yoran Heling - -Permission is hereby granted, free of charge, to any person obtaining -a copy of this software and associated documentation files (the -"Software"), to deal in the Software without restriction, including -without limitation the rights to use, copy, modify, merge, publish, -distribute, sublicense, and/or sell copies of the Software, and to -permit persons to whom the Software is furnished to do so, subject to -the following conditions: - -The above copyright notice and this permission notice shall be included -in all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, -EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF -MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. -IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY -CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, -TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE -SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + GNU AFFERO GENERAL PUBLIC LICENSE + Version 3, 19 November 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/> + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The GNU Affero General Public License is a free, copyleft license for +software and other kinds of works, specifically designed to ensure +cooperation with the community in the case of network server software. + + The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +our General Public Licenses are intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains free +software for all its users. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + + Developers that use our General Public Licenses protect your rights +with two steps: (1) assert copyright on the software, and (2) offer +you this License which gives you legal permission to copy, distribute +and/or modify the software. + + A secondary benefit of defending all users' freedom is that +improvements made in alternate versions of the program, if they +receive widespread use, become available for other developers to +incorporate. Many developers of free software are heartened and +encouraged by the resulting cooperation. However, in the case of +software used on network servers, this result may fail to come about. +The GNU General Public License permits making a modified version and +letting the public access it on a server without ever releasing its +source code to the public. + + The GNU Affero General Public License is designed specifically to +ensure that, in such cases, the modified source code becomes available +to the community. It requires the operator of a network server to +provide the source code of the modified version running there to the +users of that server. Therefore, public use of a modified version, on +a publicly accessible server, gives the public access to the source +code of the modified version. + + An older license, called the Affero General Public License and +published by Affero, was designed to accomplish similar goals. This is +a different license, not a version of the Affero GPL, but Affero has +released a new version of the Affero GPL which permits relicensing under +this license. + + The precise terms and conditions for copying, distribution and +modification follow. + + TERMS AND CONDITIONS + + 0. Definitions. + + "This License" refers to version 3 of the GNU Affero General Public License. + + "Copyright" also means copyright-like laws that apply to other kinds of +works, such as semiconductor masks. + + "The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + + To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of an +exact copy. The resulting work is called a "modified version" of the +earlier work or a work "based on" the earlier work. + + A "covered work" means either the unmodified Program or a work based +on the Program. + + To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + + To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user through +a computer network, with no transfer of a copy, is not conveying. + + An interactive user interface displays "Appropriate Legal Notices" +to the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + + 1. Source Code. + + The "source code" for a work means the preferred form of the work +for making modifications to it. "Object code" means any non-source +form of a work. + + A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + + The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + + The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + + The Corresponding Source need not include anything that users +can regenerate automatically from other parts of the Corresponding +Source. + + The Corresponding Source for a work in source code form is that +same work. + + 2. Basic Permissions. + + All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + + You may make, run and propagate covered works that you do not +convey, without conditions so long as your license otherwise remains +in force. You may convey covered works to others for the sole purpose +of having them make modifications exclusively for you, or provide you +with facilities for running those works, provided that you comply with +the terms of this License in conveying all material for which you do +not control copyright. Those thus making or running the covered works +for you must do so exclusively on your behalf, under your direction +and control, on terms that prohibit them from making any copies of +your copyrighted material outside their relationship with you. + + Conveying under any other circumstances is permitted solely under +the conditions stated below. Sublicensing is not allowed; section 10 +makes it unnecessary. + + 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + + No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + + When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such circumvention +is effected by exercising rights under this License with respect to +the covered work, and you disclaim any intention to limit operation or +modification of the work as a means of enforcing, against the work's +users, your or third parties' legal rights to forbid circumvention of +technological measures. + + 4. Conveying Verbatim Copies. + + You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + + You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + + 5. Conveying Modified Source Versions. + + You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. + + b) The work must carry prominent notices stating that it is + released under this License and any conditions added under section + 7. This requirement modifies the requirement in section 4 to + "keep intact all notices". + + c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. + + d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + + A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + + 6. Conveying Non-Source Forms. + + You may convey a covered work in object code form under the terms +of sections 4 and 5, provided that you also convey the +machine-readable Corresponding Source under the terms of this License, +in one of these ways: + + a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. + + b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the + Corresponding Source from a network server at no charge. + + c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. + + d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. + + e) Convey the object code using peer-to-peer transmission, provided + you inform other peers where the object code and Corresponding + Source of the work are being offered to the general public at no + charge under subsection 6d. + + A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + + A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, family, +or household purposes, or (2) anything designed or sold for incorporation +into a dwelling. In determining whether a product is a consumer product, +doubtful cases shall be resolved in favor of coverage. For a particular +product received by a particular user, "normally used" refers to a +typical or common use of that class of product, regardless of the status +of the particular user or of the way in which the particular user +actually uses, or expects or is expected to use, the product. A product +is a consumer product regardless of whether the product has substantial +commercial, industrial or non-consumer uses, unless such uses represent +the only significant mode of use of the product. + + "Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to install +and execute modified versions of a covered work in that User Product from +a modified version of its Corresponding Source. The information must +suffice to ensure that the continued functioning of the modified object +code is in no case prevented or interfered with solely because +modification has been made. + + If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + + The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or updates +for a work that has been modified or installed by the recipient, or for +the User Product in which it has been modified or installed. Access to a +network may be denied when the modification itself materially and +adversely affects the operation of the network or violates the rules and +protocols for communication across the network. + + Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + + 7. Additional Terms. + + "Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + + When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + + Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders of +that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or + + b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or + + c) Prohibiting misrepresentation of the origin of that material, or + requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or + + d) Limiting the use for publicity purposes of names of licensors or + authors of the material; or + + e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or + + f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions of + it) with contractual assumptions of liability to the recipient, for + any liability that these contractual assumptions directly impose on + those licensors and authors. + + All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + + If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + + Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; +the above requirements apply either way. + + 8. Termination. + + You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + + However, if you cease all violation of this License, then your +license from a particular copyright holder is reinstated (a) +provisionally, unless and until the copyright holder explicitly and +finally terminates your license, and (b) permanently, if the copyright +holder fails to notify you of the violation by some reasonable means +prior to 60 days after the cessation. + + Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + + Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + + 9. Acceptance Not Required for Having Copies. + + You are not required to accept this License in order to receive or +run a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + + 10. Automatic Licensing of Downstream Recipients. + + Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + + An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + + You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + + 11. Patents. + + A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + + A contributor's "essential patent claims" are all patent claims +owned or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + + Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + + In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + + If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + + If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + + A patent license is "discriminatory" if it does not include within +the scope of its coverage, prohibits the exercise of, or is +conditioned on the non-exercise of one or more of the rights that are +specifically granted under this License. You may not convey a covered +work if you are a party to an arrangement with a third party that is +in the business of distributing software, under which you make payment +to the third party based on the extent of your activity of conveying +the work, and under which the third party grants, to any of the +parties who would receive the covered work from you, a discriminatory +patent license (a) in connection with copies of the covered work +conveyed by you (or copies made from those copies), or (b) primarily +for and in connection with specific products or compilations that +contain the covered work, unless you entered into that arrangement, +or that patent license was granted, prior to 28 March 2007. + + Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + + 12. No Surrender of Others' Freedom. + + If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you may +not convey it at all. For example, if you agree to terms that obligate you +to collect a royalty for further conveying from those to whom you convey +the Program, the only way you could satisfy both those terms and this +License would be to refrain entirely from conveying the Program. + + 13. Remote Network Interaction; Use with the GNU General Public License. + + Notwithstanding any other provision of this License, if you modify the +Program, your modified version must prominently offer all users +interacting with it remotely through a computer network (if your version +supports such interaction) an opportunity to receive the Corresponding +Source of your version by providing access to the Corresponding Source +from a network server at no charge, through some standard or customary +means of facilitating copying of software. This Corresponding Source +shall include the Corresponding Source for any work covered by version 3 +of the GNU General Public License that is incorporated pursuant to the +following paragraph. + + Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the work with which it is combined will remain governed by version +3 of the GNU General Public License. + + 14. Revised Versions of this License. + + The Free Software Foundation may publish revised and/or new versions of +the GNU Affero General Public License from time to time. Such new versions +will be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + + Each version is given a distinguishing version number. If the +Program specifies that a certain numbered version of the GNU Affero General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU Affero General Public License, you may choose any version ever published +by the Free Software Foundation. + + If the Program specifies that a proxy can decide which future +versions of the GNU Affero General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + + Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + + 15. Disclaimer of Warranty. + + THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY +OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM +IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF +ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. Limitation of Liability. + + IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS +THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY +GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE +USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF +DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD +PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), +EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF +SUCH DAMAGES. + + 17. Interpretation of Sections 15 and 16. + + If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +state the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + <one line to give the program's name and a brief idea of what it does.> + Copyright (C) <year> <name of author> + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU Affero General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU Affero General Public License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see <http://www.gnu.org/licenses/>. + +Also add information on how to contact you by electronic and paper mail. + + If your software can interact with users remotely through a computer +network, you should also make sure that it provides a way for users to +get its source. For example, if your program is a web application, its +interface could display a "Source" link that leads users to an archive +of the code. There are many ways you could offer source, and different +solutions will be better for different programs; see section 13 for the +specific requirements. + + You should also get your employer (if you work as a programmer) or school, +if any, to sign a "copyright disclaimer" for the program, if necessary. +For more information on this, and how to apply and follow the GNU AGPL, see +<http://www.gnu.org/licenses/>. @@ -9,15 +9,16 @@ Ironically, documentation about how things work is completely lacking. - perl: A somewhat recent version (no idea which, due to my XS usage) - postgresql: Also a somewhat recent version -- rust + cargo (1.13+) +- rust: Version who-knows-which ### Web front-end -- DBI +- AnyEvent - DBD::Pg -- TUWF +- DBI - JSON::XS -- AnyEvent +- SQL::Interp +- TUWF ### Man page indexer diff --git a/indexer/Cargo.lock b/indexer/Cargo.lock index 7eada65..a614ac6 100644 --- a/indexer/Cargo.lock +++ b/indexer/Cargo.lock @@ -1,322 +1,177 @@ # This file is automatically @generated by Cargo. # It is not intended for manual editing. [[package]] -name = "adler32" -version = "1.0.3" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] name = "aho-corasick" -version = "0.7.3" +version = "0.7.18" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e37cfd5e7657ada45f742d6e99ca5788580b5c529dc78faf11ece6dc702656f" dependencies = [ - "memchr 2.2.0 (registry+https://github.com/rust-lang/crates.io-index)", + "memchr", ] [[package]] name = "ansi_term" -version = "0.11.0" +version = "0.12.1" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d52a9bb7ec0cf484c551830a7ce27bd20d67eac647e1befb56b0be4ee39a55d2" dependencies = [ - "winapi 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)", + "winapi", ] [[package]] -name = "arrayref" -version = "0.3.5" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "arrayvec" -version = "0.4.10" +name = "async-trait" +version = "0.1.52" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "061a7acccaa286c011ddc30970520b98fa40e00c9d644633fb26b5fc63a265e3" dependencies = [ - "nodrop 0.1.13 (registry+https://github.com/rust-lang/crates.io-index)", + "proc-macro2", + "quote", + "syn", ] [[package]] name = "atty" -version = "0.2.11" +version = "0.2.14" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d9b39be18770d11421cdb1b9947a45dd3f37e93092cbf377614828a319d5fee8" dependencies = [ - "libc 0.2.55 (registry+https://github.com/rust-lang/crates.io-index)", - "termion 1.5.2 (registry+https://github.com/rust-lang/crates.io-index)", - "winapi 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)", + "hermit-abi", + "libc", + "winapi", ] [[package]] name = "autocfg" -version = "0.1.4" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "backtrace" -version = "0.3.26" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "autocfg 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)", - "backtrace-sys 0.1.28 (registry+https://github.com/rust-lang/crates.io-index)", - "cfg-if 0.1.9 (registry+https://github.com/rust-lang/crates.io-index)", - "libc 0.2.55 (registry+https://github.com/rust-lang/crates.io-index)", - "rustc-demangle 0.1.14 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "backtrace-sys" -version = "0.1.28" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "cc 1.0.37 (registry+https://github.com/rust-lang/crates.io-index)", - "libc 0.2.55 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "base64" -version = "0.6.0" +version = "1.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "byteorder 1.3.1 (registry+https://github.com/rust-lang/crates.io-index)", - "safemem 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)", -] +checksum = "cdb031dd78e28731d87d56cc8ffef4a8f36ca26c38fe2de700543e627f8a464a" [[package]] name = "base64" -version = "0.10.1" +version = "0.13.0" source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "byteorder 1.3.1 (registry+https://github.com/rust-lang/crates.io-index)", -] +checksum = "904dfeac50f3cdaba28fc6f57fdcddb75f49ed61346676a78c4ffe55877802fd" [[package]] name = "bitflags" -version = "1.0.4" +version = "1.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a" [[package]] name = "block-buffer" -version = "0.3.3" +version = "0.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f1d36a02058e76b040de25a4464ba1c80935655595b661505c8b39b664828b95" dependencies = [ - "arrayref 0.3.5 (registry+https://github.com/rust-lang/crates.io-index)", - "byte-tools 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)", + "generic-array", ] [[package]] -name = "build_const" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "byte-tools" -version = "0.2.0" +name = "bumpalo" +version = "3.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f1e260c3a9040a7c19a12468758f4c16f31a81a1fe087482be9570ec864bb6c" [[package]] name = "byteorder" -version = "1.3.1" +version = "1.4.3" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "14c189c53d098945499cdfa7ecc63567cf3886b3332b312a5b4585d8d3a6a610" [[package]] name = "bytes" -version = "0.4.12" +version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "byteorder 1.3.1 (registry+https://github.com/rust-lang/crates.io-index)", - "either 1.5.2 (registry+https://github.com/rust-lang/crates.io-index)", - "iovec 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)", -] +checksum = "c4872d67bab6358e59559027aa3b9157c53d9358c51423c17554809a8858e0f8" [[package]] name = "cc" -version = "1.0.37" +version = "1.0.72" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "22a9137b95ea06864e018375b72adfb7db6e6f68cfc8df5a04d00288050485ee" [[package]] name = "cfg-if" -version = "0.1.9" +version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd" [[package]] name = "chrono" -version = "0.4.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "num-integer 0.1.41 (registry+https://github.com/rust-lang/crates.io-index)", - "num-traits 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)", - "time 0.1.42 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "clap" -version = "2.33.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "ansi_term 0.11.0 (registry+https://github.com/rust-lang/crates.io-index)", - "atty 0.2.11 (registry+https://github.com/rust-lang/crates.io-index)", - "bitflags 1.0.4 (registry+https://github.com/rust-lang/crates.io-index)", - "strsim 0.8.0 (registry+https://github.com/rust-lang/crates.io-index)", - "textwrap 0.11.0 (registry+https://github.com/rust-lang/crates.io-index)", - "unicode-width 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)", - "vec_map 0.8.1 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "cloudabi" -version = "0.0.3" +version = "0.4.19" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "670ad68c9088c2a963aaa298cb369688cf3f9465ce5e2d4ca10e6e0098a1ce73" dependencies = [ - "bitflags 1.0.4 (registry+https://github.com/rust-lang/crates.io-index)", + "libc", + "num-integer", + "num-traits", + "time", + "winapi", ] [[package]] -name = "constant_time_eq" -version = "0.1.3" +name = "chunked_transfer" +version = "1.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fff857943da45f546682664a79488be82e69e43c1a7a2307679ab9afb3a66d2e" [[package]] -name = "cookie" -version = "0.12.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "time 0.1.42 (registry+https://github.com/rust-lang/crates.io-index)", - "url 1.7.2 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "cookie_store" -version = "0.7.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "cookie 0.12.0 (registry+https://github.com/rust-lang/crates.io-index)", - "failure 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)", - "idna 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)", - "log 0.4.6 (registry+https://github.com/rust-lang/crates.io-index)", - "publicsuffix 1.5.2 (registry+https://github.com/rust-lang/crates.io-index)", - "serde 1.0.91 (registry+https://github.com/rust-lang/crates.io-index)", - "serde_json 1.0.39 (registry+https://github.com/rust-lang/crates.io-index)", - "time 0.1.42 (registry+https://github.com/rust-lang/crates.io-index)", - "try_from 0.3.2 (registry+https://github.com/rust-lang/crates.io-index)", - "url 1.7.2 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "core-foundation" -version = "0.6.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "core-foundation-sys 0.6.2 (registry+https://github.com/rust-lang/crates.io-index)", - "libc 0.2.55 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "core-foundation-sys" -version = "0.6.2" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "crc" -version = "1.8.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "build_const 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "crc32fast" -version = "1.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "cfg-if 0.1.9 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "crossbeam-deque" -version = "0.7.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "crossbeam-epoch 0.7.1 (registry+https://github.com/rust-lang/crates.io-index)", - "crossbeam-utils 0.6.5 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "crossbeam-epoch" -version = "0.7.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "arrayvec 0.4.10 (registry+https://github.com/rust-lang/crates.io-index)", - "cfg-if 0.1.9 (registry+https://github.com/rust-lang/crates.io-index)", - "crossbeam-utils 0.6.5 (registry+https://github.com/rust-lang/crates.io-index)", - "lazy_static 1.3.0 (registry+https://github.com/rust-lang/crates.io-index)", - "memoffset 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)", - "scopeguard 0.3.3 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "crossbeam-queue" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "crossbeam-utils 0.6.5 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "crossbeam-utils" -version = "0.6.5" +name = "clap" +version = "2.34.0" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a0610544180c38b88101fecf2dd634b174a62eef6946f84dfc6a7127512b381c" dependencies = [ - "cfg-if 0.1.9 (registry+https://github.com/rust-lang/crates.io-index)", - "lazy_static 1.3.0 (registry+https://github.com/rust-lang/crates.io-index)", + "ansi_term", + "atty", + "bitflags", + "strsim", + "textwrap", + "unicode-width", + "vec_map", ] [[package]] -name = "crypto-mac" -version = "0.5.2" +name = "cpufeatures" +version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "95059428f66df56b63431fdb4e1947ed2190586af5c5a8a8b71122bdf5a7f469" dependencies = [ - "constant_time_eq 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)", - "generic-array 0.9.0 (registry+https://github.com/rust-lang/crates.io-index)", + "libc", ] [[package]] -name = "derive_more" -version = "0.14.0" +name = "crypto-common" +version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "567569e659735adb39ff2d4c20600f7cd78be5471f8c58ab162bce3c03fdbc5f" dependencies = [ - "proc-macro2 0.4.30 (registry+https://github.com/rust-lang/crates.io-index)", - "quote 0.6.12 (registry+https://github.com/rust-lang/crates.io-index)", - "rustc_version 0.2.3 (registry+https://github.com/rust-lang/crates.io-index)", - "syn 0.15.34 (registry+https://github.com/rust-lang/crates.io-index)", + "generic-array", ] [[package]] name = "digest" -version = "0.7.6" +version = "0.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8549e6bfdecd113b7e221fe60b433087f6957387a20f8118ebca9b12af19143d" dependencies = [ - "generic-array 0.9.0 (registry+https://github.com/rust-lang/crates.io-index)", + "block-buffer", + "crypto-common", + "generic-array", + "subtle", ] [[package]] -name = "dtoa" -version = "0.4.4" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "either" -version = "1.5.2" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] name = "encoding" version = "0.3.0-dev" source = "git+https://github.com/lifthrasiir/rust-encoding#eb3d3c307df864f6a25e2ca16d49703e5d963ec5" dependencies = [ - "encoding-index-japanese 1.20141219.6 (git+https://github.com/lifthrasiir/rust-encoding)", - "encoding-index-korean 1.20141219.6 (git+https://github.com/lifthrasiir/rust-encoding)", - "encoding-index-simpchinese 1.20160120.0 (git+https://github.com/lifthrasiir/rust-encoding)", - "encoding-index-singlebyte 1.20160120.0 (git+https://github.com/lifthrasiir/rust-encoding)", - "encoding-index-tradchinese 1.20141219.6 (git+https://github.com/lifthrasiir/rust-encoding)", - "encoding-types 0.2.0 (git+https://github.com/lifthrasiir/rust-encoding)", + "encoding-index-japanese", + "encoding-index-korean", + "encoding-index-simpchinese", + "encoding-index-singlebyte", + "encoding-index-tradchinese", + "encoding-types", ] [[package]] @@ -324,7 +179,7 @@ name = "encoding-index-japanese" version = "1.20141219.6" source = "git+https://github.com/lifthrasiir/rust-encoding#eb3d3c307df864f6a25e2ca16d49703e5d963ec5" dependencies = [ - "encoding_index_tests 0.1.5 (git+https://github.com/lifthrasiir/rust-encoding)", + "encoding_index_tests", ] [[package]] @@ -332,7 +187,7 @@ name = "encoding-index-korean" version = "1.20141219.6" source = "git+https://github.com/lifthrasiir/rust-encoding#eb3d3c307df864f6a25e2ca16d49703e5d963ec5" dependencies = [ - "encoding_index_tests 0.1.5 (git+https://github.com/lifthrasiir/rust-encoding)", + "encoding_index_tests", ] [[package]] @@ -340,7 +195,7 @@ name = "encoding-index-simpchinese" version = "1.20160120.0" source = "git+https://github.com/lifthrasiir/rust-encoding#eb3d3c307df864f6a25e2ca16d49703e5d963ec5" dependencies = [ - "encoding_index_tests 0.1.5 (git+https://github.com/lifthrasiir/rust-encoding)", + "encoding_index_tests", ] [[package]] @@ -348,7 +203,7 @@ name = "encoding-index-singlebyte" version = "1.20160120.0" source = "git+https://github.com/lifthrasiir/rust-encoding#eb3d3c307df864f6a25e2ca16d49703e5d963ec5" dependencies = [ - "encoding_index_tests 0.1.5 (git+https://github.com/lifthrasiir/rust-encoding)", + "encoding_index_tests", ] [[package]] @@ -356,7 +211,7 @@ name = "encoding-index-tradchinese" version = "1.20141219.6" source = "git+https://github.com/lifthrasiir/rust-encoding#eb3d3c307df864f6a25e2ca16d49703e5d963ec5" dependencies = [ - "encoding_index_tests 0.1.5 (git+https://github.com/lifthrasiir/rust-encoding)", + "encoding_index_tests", ] [[package]] @@ -370,1656 +225,954 @@ version = "0.1.5" source = "git+https://github.com/lifthrasiir/rust-encoding#eb3d3c307df864f6a25e2ca16d49703e5d963ec5" [[package]] -name = "encoding_rs" -version = "0.8.17" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "cfg-if 0.1.9 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] name = "env_logger" -version = "0.6.1" +version = "0.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b2cf0344971ee6c64c31be0d530793fba457d322dfec2810c453d0ef228f9c3" dependencies = [ - "atty 0.2.11 (registry+https://github.com/rust-lang/crates.io-index)", - "humantime 1.2.0 (registry+https://github.com/rust-lang/crates.io-index)", - "log 0.4.6 (registry+https://github.com/rust-lang/crates.io-index)", - "regex 1.1.6 (registry+https://github.com/rust-lang/crates.io-index)", - "termcolor 1.0.4 (registry+https://github.com/rust-lang/crates.io-index)", + "atty", + "humantime", + "log", + "regex", + "termcolor", ] [[package]] -name = "error-chain" -version = "0.12.1" +name = "fallible-iterator" +version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "backtrace 0.3.26 (registry+https://github.com/rust-lang/crates.io-index)", - "version_check 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)", -] +checksum = "4443176a9f2c162692bd3d352d745ef9413eec5782a80d8fd6f8a1ac692a07f7" [[package]] -name = "failure" -version = "0.1.5" +name = "form_urlencoded" +version = "1.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5fc25a87fa4fd2094bffb06925852034d90a17f0d1e05197d4956d3555752191" dependencies = [ - "backtrace 0.3.26 (registry+https://github.com/rust-lang/crates.io-index)", - "failure_derive 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)", + "matches", + "percent-encoding", ] [[package]] -name = "failure_derive" -version = "0.1.5" +name = "futures" +version = "0.3.18" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8cd0210d8c325c245ff06fd95a3b13689a1a276ac8cfa8e8720cb840bfb84b9e" dependencies = [ - "proc-macro2 0.4.30 (registry+https://github.com/rust-lang/crates.io-index)", - "quote 0.6.12 (registry+https://github.com/rust-lang/crates.io-index)", - "syn 0.15.34 (registry+https://github.com/rust-lang/crates.io-index)", - "synstructure 0.10.2 (registry+https://github.com/rust-lang/crates.io-index)", + "futures-channel", + "futures-core", + "futures-executor", + "futures-io", + "futures-sink", + "futures-task", + "futures-util", ] [[package]] -name = "fake-simd" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "fallible-iterator" -version = "0.1.6" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "flate2" -version = "1.0.7" +name = "futures-channel" +version = "0.3.18" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7fc8cd39e3dbf865f7340dce6a2d401d24fd37c6fe6c4f0ee0de8bfca2252d27" dependencies = [ - "crc32fast 1.2.0 (registry+https://github.com/rust-lang/crates.io-index)", - "libc 0.2.55 (registry+https://github.com/rust-lang/crates.io-index)", - "miniz_oxide_c_api 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)", + "futures-core", + "futures-sink", ] [[package]] -name = "fnv" -version = "1.0.6" +name = "futures-core" +version = "0.3.18" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "629316e42fe7c2a0b9a65b47d159ceaa5453ab14e8f0a3c5eedbb8cd55b4a445" [[package]] -name = "foreign-types" -version = "0.3.2" +name = "futures-executor" +version = "0.3.18" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7b808bf53348a36cab739d7e04755909b9fcaaa69b7d7e588b37b6ec62704c97" dependencies = [ - "foreign-types-shared 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)", + "futures-core", + "futures-task", + "futures-util", ] [[package]] -name = "foreign-types-shared" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "fuchsia-cprng" -version = "0.1.1" +name = "futures-io" +version = "0.3.18" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e481354db6b5c353246ccf6a728b0c5511d752c08da7260546fc0933869daa11" [[package]] -name = "fuchsia-zircon" -version = "0.3.3" +name = "futures-macro" +version = "0.3.18" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a89f17b21645bc4ed773c69af9c9a0effd4a3f1a3876eadd453469f8854e7fdd" dependencies = [ - "bitflags 1.0.4 (registry+https://github.com/rust-lang/crates.io-index)", - "fuchsia-zircon-sys 0.3.3 (registry+https://github.com/rust-lang/crates.io-index)", + "proc-macro2", + "quote", + "syn", ] [[package]] -name = "fuchsia-zircon-sys" -version = "0.3.3" +name = "futures-sink" +version = "0.3.18" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "996c6442437b62d21a32cd9906f9c41e7dc1e19a9579843fad948696769305af" [[package]] -name = "futures" -version = "0.1.27" +name = "futures-task" +version = "0.3.18" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dabf1872aaab32c886832f2276d2f5399887e2bd613698a02359e4ea83f8de12" [[package]] -name = "futures-cpupool" -version = "0.1.8" +name = "futures-util" +version = "0.3.18" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41d22213122356472061ac0f1ab2cee28d2bac8491410fd68c2af53d1cedb83e" dependencies = [ - "futures 0.1.27 (registry+https://github.com/rust-lang/crates.io-index)", - "num_cpus 1.10.0 (registry+https://github.com/rust-lang/crates.io-index)", + "futures-channel", + "futures-core", + "futures-io", + "futures-macro", + "futures-sink", + "futures-task", + "memchr", + "pin-project-lite", + "pin-utils", + "slab", ] [[package]] name = "generic-array" -version = "0.9.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "typenum 1.10.0 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "h2" -version = "0.1.20" +version = "0.14.4" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "501466ecc8a30d1d3b7fc9229b122b2ce8ed6e9d9223f1138d4babb253e51817" dependencies = [ - "byteorder 1.3.1 (registry+https://github.com/rust-lang/crates.io-index)", - "bytes 0.4.12 (registry+https://github.com/rust-lang/crates.io-index)", - "fnv 1.0.6 (registry+https://github.com/rust-lang/crates.io-index)", - "futures 0.1.27 (registry+https://github.com/rust-lang/crates.io-index)", - "http 0.1.17 (registry+https://github.com/rust-lang/crates.io-index)", - "indexmap 1.0.2 (registry+https://github.com/rust-lang/crates.io-index)", - "log 0.4.6 (registry+https://github.com/rust-lang/crates.io-index)", - "slab 0.4.2 (registry+https://github.com/rust-lang/crates.io-index)", - "string 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio-io 0.1.12 (registry+https://github.com/rust-lang/crates.io-index)", + "typenum", + "version_check", ] [[package]] -name = "hex" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "hmac" -version = "0.5.0" +name = "getrandom" +version = "0.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7fcd999463524c52659517fe2cea98493cfe485d10565e7b0fb07dbba7ad2753" dependencies = [ - "crypto-mac 0.5.2 (registry+https://github.com/rust-lang/crates.io-index)", - "digest 0.7.6 (registry+https://github.com/rust-lang/crates.io-index)", + "cfg-if", + "libc", + "wasi", ] [[package]] -name = "http" -version = "0.1.17" +name = "hermit-abi" +version = "0.1.19" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "62b467343b94ba476dcb2500d242dadbb39557df889310ac77c5d99100aaac33" dependencies = [ - "bytes 0.4.12 (registry+https://github.com/rust-lang/crates.io-index)", - "fnv 1.0.6 (registry+https://github.com/rust-lang/crates.io-index)", - "itoa 0.4.4 (registry+https://github.com/rust-lang/crates.io-index)", + "libc", ] [[package]] -name = "http-body" -version = "0.1.0" +name = "hmac" +version = "0.12.0" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ddca131f3e7f2ce2df364b57949a9d47915cfbd35e46cfee355ccebbf794d6a2" dependencies = [ - "bytes 0.4.12 (registry+https://github.com/rust-lang/crates.io-index)", - "futures 0.1.27 (registry+https://github.com/rust-lang/crates.io-index)", - "http 0.1.17 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio-buf 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)", + "digest", ] [[package]] -name = "httparse" -version = "1.3.3" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] name = "humantime" -version = "1.2.0" +version = "2.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "quick-error 1.2.2 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "hyper" -version = "0.12.29" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "bytes 0.4.12 (registry+https://github.com/rust-lang/crates.io-index)", - "futures 0.1.27 (registry+https://github.com/rust-lang/crates.io-index)", - "futures-cpupool 0.1.8 (registry+https://github.com/rust-lang/crates.io-index)", - "h2 0.1.20 (registry+https://github.com/rust-lang/crates.io-index)", - "http 0.1.17 (registry+https://github.com/rust-lang/crates.io-index)", - "http-body 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)", - "httparse 1.3.3 (registry+https://github.com/rust-lang/crates.io-index)", - "iovec 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)", - "itoa 0.4.4 (registry+https://github.com/rust-lang/crates.io-index)", - "log 0.4.6 (registry+https://github.com/rust-lang/crates.io-index)", - "net2 0.2.33 (registry+https://github.com/rust-lang/crates.io-index)", - "rustc_version 0.2.3 (registry+https://github.com/rust-lang/crates.io-index)", - "time 0.1.42 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio 0.1.20 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio-buf 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio-executor 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio-io 0.1.12 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio-reactor 0.1.9 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio-tcp 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio-threadpool 0.1.14 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio-timer 0.2.11 (registry+https://github.com/rust-lang/crates.io-index)", - "want 0.0.6 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "hyper-tls" -version = "0.3.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "bytes 0.4.12 (registry+https://github.com/rust-lang/crates.io-index)", - "futures 0.1.27 (registry+https://github.com/rust-lang/crates.io-index)", - "hyper 0.12.29 (registry+https://github.com/rust-lang/crates.io-index)", - "native-tls 0.2.3 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio-io 0.1.12 (registry+https://github.com/rust-lang/crates.io-index)", -] +checksum = "9a3a5bfb195931eeb336b2a7b4d761daec841b97f947d34394601737a7bba5e4" [[package]] name = "idna" -version = "0.1.5" +version = "0.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "418a0a6fab821475f634efe3ccc45c013f742efe03d853e8d3355d5cb850ecf8" dependencies = [ - "matches 0.1.8 (registry+https://github.com/rust-lang/crates.io-index)", - "unicode-bidi 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)", - "unicode-normalization 0.1.8 (registry+https://github.com/rust-lang/crates.io-index)", + "matches", + "unicode-bidi", + "unicode-normalization", ] [[package]] name = "indexer" version = "0.1.0" dependencies = [ - "chrono 0.4.6 (registry+https://github.com/rust-lang/crates.io-index)", - "clap 2.33.0 (registry+https://github.com/rust-lang/crates.io-index)", - "encoding 0.3.0-dev (git+https://github.com/lifthrasiir/rust-encoding)", - "env_logger 0.6.1 (registry+https://github.com/rust-lang/crates.io-index)", - "lazy_static 1.3.0 (registry+https://github.com/rust-lang/crates.io-index)", - "libarchive3-sys 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)", - "libc 0.2.55 (registry+https://github.com/rust-lang/crates.io-index)", - "log 0.4.6 (registry+https://github.com/rust-lang/crates.io-index)", - "postgres 0.15.2 (registry+https://github.com/rust-lang/crates.io-index)", - "quick-xml 0.14.0 (registry+https://github.com/rust-lang/crates.io-index)", - "regex 1.1.6 (registry+https://github.com/rust-lang/crates.io-index)", - "reqwest 0.9.17 (registry+https://github.com/rust-lang/crates.io-index)", - "ring 0.14.6 (registry+https://github.com/rust-lang/crates.io-index)", - "url 1.7.2 (registry+https://github.com/rust-lang/crates.io-index)", + "chrono", + "clap", + "encoding", + "env_logger", + "lazy_static", + "libarchive3-sys", + "libc", + "log", + "percent-encoding", + "postgres", + "quick-xml", + "regex", + "ring", + "ureq", ] [[package]] -name = "indexmap" -version = "1.0.2" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "iovec" -version = "0.1.2" +name = "instant" +version = "0.1.12" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7a5bbe824c507c5da5956355e86a746d82e0e1464f65d862cc5e71da70e94b2c" dependencies = [ - "libc 0.2.55 (registry+https://github.com/rust-lang/crates.io-index)", - "winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)", + "cfg-if", ] [[package]] -name = "itoa" -version = "0.4.4" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "kernel32-sys" -version = "0.2.2" +name = "js-sys" +version = "0.3.55" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7cc9ffccd38c451a86bf13657df244e9c3f37493cce8e5e21e940963777acc84" dependencies = [ - "winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)", - "winapi-build 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)", + "wasm-bindgen", ] [[package]] name = "lazy_static" -version = "1.3.0" +version = "1.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e2abad23fbc42b3700f2f279844dc832adb2b2eb069b2df918f455c4e18cc646" [[package]] name = "libarchive3-sys" version = "0.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3cd3beae8f59a4c7a806523269b5392037577c150446e88d684dfa6de6031ca7" dependencies = [ - "libc 0.2.55 (registry+https://github.com/rust-lang/crates.io-index)", - "pkg-config 0.3.14 (registry+https://github.com/rust-lang/crates.io-index)", + "libc", + "pkg-config", ] [[package]] name = "libc" -version = "0.2.55" +version = "0.2.109" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f98a04dce437184842841303488f70d0188c5f51437d2a834dc097eafa909a01" [[package]] name = "lock_api" -version = "0.1.5" +version = "0.4.5" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "712a4d093c9976e24e7dbca41db895dabcbac38eb5f4045393d17a95bdfb1109" dependencies = [ - "owning_ref 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)", - "scopeguard 0.3.3 (registry+https://github.com/rust-lang/crates.io-index)", + "scopeguard", ] [[package]] name = "log" -version = "0.4.6" +version = "0.4.14" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "51b9bbe6c47d51fc3e1a9b945965946b4c44142ab8792c50835a980d362c2710" dependencies = [ - "cfg-if 0.1.9 (registry+https://github.com/rust-lang/crates.io-index)", + "cfg-if", ] [[package]] name = "matches" -version = "0.1.8" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "md5" -version = "0.3.8" +version = "0.1.9" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a3e378b66a060d48947b590737b30a1be76706c8dd7b8ba0f2fe3989c68a853f" [[package]] -name = "memchr" -version = "1.0.2" +name = "md-5" +version = "0.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6a38fc55c8bbc10058782919516f88826e70320db6d206aebc49611d24216ae" dependencies = [ - "libc 0.2.55 (registry+https://github.com/rust-lang/crates.io-index)", + "digest", ] [[package]] name = "memchr" -version = "2.2.0" +version = "2.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "memoffset" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "mime" -version = "0.3.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "unicase 2.4.0 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "mime_guess" -version = "2.0.0-alpha.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "mime 0.3.13 (registry+https://github.com/rust-lang/crates.io-index)", - "phf 0.7.24 (registry+https://github.com/rust-lang/crates.io-index)", - "phf_codegen 0.7.24 (registry+https://github.com/rust-lang/crates.io-index)", - "unicase 1.4.2 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "miniz_oxide" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "adler32 1.0.3 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "miniz_oxide_c_api" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "cc 1.0.37 (registry+https://github.com/rust-lang/crates.io-index)", - "crc 1.8.1 (registry+https://github.com/rust-lang/crates.io-index)", - "libc 0.2.55 (registry+https://github.com/rust-lang/crates.io-index)", - "miniz_oxide 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)", -] +checksum = "308cc39be01b73d0d18f82a0e7b2a3df85245f84af96fdddc5d202d27e47b86a" [[package]] name = "mio" -version = "0.6.18" +version = "0.7.14" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8067b404fe97c70829f082dec8bcf4f71225d7eaea1d8645349cb76fa06205cc" dependencies = [ - "fuchsia-zircon 0.3.3 (registry+https://github.com/rust-lang/crates.io-index)", - "fuchsia-zircon-sys 0.3.3 (registry+https://github.com/rust-lang/crates.io-index)", - "iovec 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)", - "kernel32-sys 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)", - "libc 0.2.55 (registry+https://github.com/rust-lang/crates.io-index)", - "log 0.4.6 (registry+https://github.com/rust-lang/crates.io-index)", - "miow 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)", - "net2 0.2.33 (registry+https://github.com/rust-lang/crates.io-index)", - "slab 0.4.2 (registry+https://github.com/rust-lang/crates.io-index)", - "winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)", + "libc", + "log", + "miow", + "ntapi", + "winapi", ] [[package]] name = "miow" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "kernel32-sys 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)", - "net2 0.2.33 (registry+https://github.com/rust-lang/crates.io-index)", - "winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)", - "ws2_32-sys 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "native-tls" -version = "0.2.3" +version = "0.3.7" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b9f1c5b025cda876f66ef43a113f91ebc9f4ccef34843000e0adf6ebbab84e21" dependencies = [ - "lazy_static 1.3.0 (registry+https://github.com/rust-lang/crates.io-index)", - "libc 0.2.55 (registry+https://github.com/rust-lang/crates.io-index)", - "log 0.4.6 (registry+https://github.com/rust-lang/crates.io-index)", - "openssl 0.10.23 (registry+https://github.com/rust-lang/crates.io-index)", - "openssl-probe 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)", - "openssl-sys 0.9.47 (registry+https://github.com/rust-lang/crates.io-index)", - "schannel 0.1.15 (registry+https://github.com/rust-lang/crates.io-index)", - "security-framework 0.3.1 (registry+https://github.com/rust-lang/crates.io-index)", - "security-framework-sys 0.3.1 (registry+https://github.com/rust-lang/crates.io-index)", - "tempfile 3.0.8 (registry+https://github.com/rust-lang/crates.io-index)", + "winapi", ] [[package]] -name = "net2" -version = "0.2.33" +name = "ntapi" +version = "0.3.6" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f6bb902e437b6d86e03cce10a7e2af662292c5dfef23b65899ea3ac9354ad44" dependencies = [ - "cfg-if 0.1.9 (registry+https://github.com/rust-lang/crates.io-index)", - "libc 0.2.55 (registry+https://github.com/rust-lang/crates.io-index)", - "winapi 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)", + "winapi", ] [[package]] -name = "nodrop" -version = "0.1.13" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] name = "num-integer" -version = "0.1.41" +version = "0.1.44" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d2cc698a63b549a70bc047073d2949cce27cd1c7b0a4a862d08a8031bc2801db" dependencies = [ - "autocfg 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)", - "num-traits 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)", + "autocfg", + "num-traits", ] [[package]] name = "num-traits" -version = "0.2.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "autocfg 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "num_cpus" -version = "1.10.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "libc 0.2.55 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "numtoa" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "openssl" -version = "0.10.23" +version = "0.2.14" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a64b1ec5cda2586e284722486d802acf1f7dbdc623e2bfc57e65ca1cd099290" dependencies = [ - "bitflags 1.0.4 (registry+https://github.com/rust-lang/crates.io-index)", - "cfg-if 0.1.9 (registry+https://github.com/rust-lang/crates.io-index)", - "foreign-types 0.3.2 (registry+https://github.com/rust-lang/crates.io-index)", - "lazy_static 1.3.0 (registry+https://github.com/rust-lang/crates.io-index)", - "libc 0.2.55 (registry+https://github.com/rust-lang/crates.io-index)", - "openssl-sys 0.9.47 (registry+https://github.com/rust-lang/crates.io-index)", + "autocfg", ] [[package]] -name = "openssl-probe" -version = "0.1.2" +name = "once_cell" +version = "1.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "openssl-sys" -version = "0.9.47" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "autocfg 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)", - "cc 1.0.37 (registry+https://github.com/rust-lang/crates.io-index)", - "libc 0.2.55 (registry+https://github.com/rust-lang/crates.io-index)", - "pkg-config 0.3.14 (registry+https://github.com/rust-lang/crates.io-index)", - "vcpkg 0.2.6 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "owning_ref" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "stable_deref_trait 1.1.1 (registry+https://github.com/rust-lang/crates.io-index)", -] +checksum = "692fcb63b64b1758029e0a96ee63e049ce8c5948587f2f7208df04625e5f6b56" [[package]] name = "parking_lot" -version = "0.7.1" +version = "0.11.2" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7d17b78036a60663b797adeaee46f5c9dfebb86948d1255007a1d6be0271ff99" dependencies = [ - "lock_api 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)", - "parking_lot_core 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)", + "instant", + "lock_api", + "parking_lot_core", ] [[package]] name = "parking_lot_core" -version = "0.4.0" +version = "0.8.5" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d76e8e1493bcac0d2766c42737f34458f1c8c50c0d23bcb24ea953affb273216" dependencies = [ - "libc 0.2.55 (registry+https://github.com/rust-lang/crates.io-index)", - "rand 0.6.5 (registry+https://github.com/rust-lang/crates.io-index)", - "rustc_version 0.2.3 (registry+https://github.com/rust-lang/crates.io-index)", - "smallvec 0.6.9 (registry+https://github.com/rust-lang/crates.io-index)", - "winapi 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)", + "cfg-if", + "instant", + "libc", + "redox_syscall", + "smallvec", + "winapi", ] [[package]] name = "percent-encoding" -version = "1.0.1" +version = "2.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d4fd5641d01c8f18a23da7b6fe29298ff4b55afcccdf78973b24cf3175fee32e" [[package]] name = "phf" -version = "0.7.24" +version = "0.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b9fc3db1018c4b59d7d582a739436478b6035138b6aecbce989fc91c3e98409f" dependencies = [ - "phf_shared 0.7.24 (registry+https://github.com/rust-lang/crates.io-index)", + "phf_shared", ] [[package]] -name = "phf_codegen" -version = "0.7.24" +name = "phf_shared" +version = "0.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6796ad771acdc0123d2a88dc428b5e38ef24456743ddb1744ed628f9815c096" dependencies = [ - "phf_generator 0.7.24 (registry+https://github.com/rust-lang/crates.io-index)", - "phf_shared 0.7.24 (registry+https://github.com/rust-lang/crates.io-index)", + "siphasher", ] [[package]] -name = "phf_generator" -version = "0.7.24" +name = "pin-project-lite" +version = "0.2.7" source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "phf_shared 0.7.24 (registry+https://github.com/rust-lang/crates.io-index)", - "rand 0.6.5 (registry+https://github.com/rust-lang/crates.io-index)", -] +checksum = "8d31d11c69a6b52a174b42bdc0c30e5e11670f90788b2c471c31c1d17d449443" [[package]] -name = "phf_shared" -version = "0.7.24" +name = "pin-utils" +version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "siphasher 0.2.3 (registry+https://github.com/rust-lang/crates.io-index)", - "unicase 1.4.2 (registry+https://github.com/rust-lang/crates.io-index)", -] +checksum = "8b870d8c151b6f2fb93e84a13146138f05d02ed11c7e7c54f8826aaaf7c9f184" [[package]] name = "pkg-config" -version = "0.3.14" +version = "0.3.24" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "58893f751c9b0412871a09abd62ecd2a00298c6c83befa223ef98c52aef40cbe" [[package]] name = "postgres" -version = "0.15.2" +version = "0.19.2" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "eb76d6535496f633fa799bb872ffb4790e9cbdedda9d35564ca0252f930c0dd5" dependencies = [ - "bytes 0.4.12 (registry+https://github.com/rust-lang/crates.io-index)", - "fallible-iterator 0.1.6 (registry+https://github.com/rust-lang/crates.io-index)", - "log 0.4.6 (registry+https://github.com/rust-lang/crates.io-index)", - "postgres-protocol 0.3.2 (registry+https://github.com/rust-lang/crates.io-index)", - "postgres-shared 0.4.2 (registry+https://github.com/rust-lang/crates.io-index)", - "socket2 0.3.9 (registry+https://github.com/rust-lang/crates.io-index)", + "bytes", + "fallible-iterator", + "futures", + "log", + "tokio", + "tokio-postgres", ] [[package]] name = "postgres-protocol" -version = "0.3.2" +version = "0.6.3" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "79ec03bce71f18b4a27c4c64c6ba2ddf74686d69b91d8714fb32ead3adaed713" dependencies = [ - "base64 0.6.0 (registry+https://github.com/rust-lang/crates.io-index)", - "byteorder 1.3.1 (registry+https://github.com/rust-lang/crates.io-index)", - "bytes 0.4.12 (registry+https://github.com/rust-lang/crates.io-index)", - "fallible-iterator 0.1.6 (registry+https://github.com/rust-lang/crates.io-index)", - "generic-array 0.9.0 (registry+https://github.com/rust-lang/crates.io-index)", - "hmac 0.5.0 (registry+https://github.com/rust-lang/crates.io-index)", - "md5 0.3.8 (registry+https://github.com/rust-lang/crates.io-index)", - "memchr 1.0.2 (registry+https://github.com/rust-lang/crates.io-index)", - "rand 0.3.23 (registry+https://github.com/rust-lang/crates.io-index)", - "sha2 0.7.1 (registry+https://github.com/rust-lang/crates.io-index)", - "stringprep 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)", + "base64", + "byteorder", + "bytes", + "fallible-iterator", + "hmac", + "md-5", + "memchr", + "rand", + "sha2", + "stringprep", ] [[package]] -name = "postgres-shared" -version = "0.4.2" +name = "postgres-types" +version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "04619f94ba0cc80999f4fc7073607cb825bc739a883cb6d20900fc5e009d6b0d" dependencies = [ - "fallible-iterator 0.1.6 (registry+https://github.com/rust-lang/crates.io-index)", - "hex 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)", - "phf 0.7.24 (registry+https://github.com/rust-lang/crates.io-index)", - "postgres-protocol 0.3.2 (registry+https://github.com/rust-lang/crates.io-index)", + "bytes", + "fallible-iterator", + "postgres-protocol", ] [[package]] -name = "proc-macro2" -version = "0.4.30" +name = "ppv-lite86" +version = "0.2.15" source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "unicode-xid 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)", -] +checksum = "ed0cfbc8191465bed66e1718596ee0b0b35d5ee1f41c5df2189d0fe8bde535ba" [[package]] -name = "publicsuffix" -version = "1.5.2" +name = "proc-macro2" +version = "1.0.33" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fb37d2df5df740e582f28f8560cf425f52bb267d872fe58358eadb554909f07a" dependencies = [ - "error-chain 0.12.1 (registry+https://github.com/rust-lang/crates.io-index)", - "idna 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)", - "lazy_static 1.3.0 (registry+https://github.com/rust-lang/crates.io-index)", - "regex 1.1.6 (registry+https://github.com/rust-lang/crates.io-index)", - "url 1.7.2 (registry+https://github.com/rust-lang/crates.io-index)", + "unicode-xid", ] [[package]] -name = "quick-error" -version = "1.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] name = "quick-xml" -version = "0.14.0" +version = "0.22.0" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8533f14c8382aaad0d592c812ac3b826162128b65662331e1127b45c3d18536b" dependencies = [ - "derive_more 0.14.0 (registry+https://github.com/rust-lang/crates.io-index)", - "encoding_rs 0.8.17 (registry+https://github.com/rust-lang/crates.io-index)", - "failure 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)", - "log 0.4.6 (registry+https://github.com/rust-lang/crates.io-index)", - "memchr 2.2.0 (registry+https://github.com/rust-lang/crates.io-index)", + "memchr", ] [[package]] name = "quote" -version = "0.6.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "proc-macro2 0.4.30 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "rand" -version = "0.3.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "libc 0.2.55 (registry+https://github.com/rust-lang/crates.io-index)", - "rand 0.4.6 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "rand" -version = "0.4.6" +version = "1.0.10" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "38bc8cc6a5f2e3655e0899c1b848643b2562f853f114bfec7be120678e3ace05" dependencies = [ - "fuchsia-cprng 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)", - "libc 0.2.55 (registry+https://github.com/rust-lang/crates.io-index)", - "rand_core 0.3.1 (registry+https://github.com/rust-lang/crates.io-index)", - "rdrand 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)", - "winapi 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)", + "proc-macro2", ] [[package]] name = "rand" -version = "0.6.5" +version = "0.8.4" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2e7573632e6454cf6b99d7aac4ccca54be06da05aca2ef7423d22d27d4d4bcd8" dependencies = [ - "autocfg 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)", - "libc 0.2.55 (registry+https://github.com/rust-lang/crates.io-index)", - "rand_chacha 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)", - "rand_core 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)", - "rand_hc 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)", - "rand_isaac 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)", - "rand_jitter 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)", - "rand_os 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)", - "rand_pcg 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)", - "rand_xorshift 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)", - "winapi 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)", + "libc", + "rand_chacha", + "rand_core", + "rand_hc", ] [[package]] name = "rand_chacha" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "autocfg 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)", - "rand_core 0.3.1 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "rand_core" version = "0.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88" dependencies = [ - "rand_core 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)", + "ppv-lite86", + "rand_core", ] [[package]] name = "rand_core" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "rand_hc" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "rand_core 0.3.1 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "rand_isaac" -version = "0.1.1" +version = "0.6.3" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d34f1408f55294453790c48b2f1ebbb1c5b4b7563eb1f418bcfcfdbb06ebb4e7" dependencies = [ - "rand_core 0.3.1 (registry+https://github.com/rust-lang/crates.io-index)", + "getrandom", ] [[package]] -name = "rand_jitter" -version = "0.1.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "libc 0.2.55 (registry+https://github.com/rust-lang/crates.io-index)", - "rand_core 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)", - "winapi 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "rand_os" -version = "0.1.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "cloudabi 0.0.3 (registry+https://github.com/rust-lang/crates.io-index)", - "fuchsia-cprng 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)", - "libc 0.2.55 (registry+https://github.com/rust-lang/crates.io-index)", - "rand_core 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)", - "rdrand 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)", - "winapi 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "rand_pcg" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "autocfg 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)", - "rand_core 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "rand_xorshift" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "rand_core 0.3.1 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "rdrand" -version = "0.4.0" +name = "rand_hc" +version = "0.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d51e9f596de227fda2ea6c84607f5558e196eeaf43c986b724ba4fb8fdf497e7" dependencies = [ - "rand_core 0.3.1 (registry+https://github.com/rust-lang/crates.io-index)", + "rand_core", ] [[package]] name = "redox_syscall" -version = "0.1.54" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "redox_termios" -version = "0.1.1" +version = "0.2.10" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8383f39639269cde97d255a32bdb68c047337295414940c68bdd30c2e13203ff" dependencies = [ - "redox_syscall 0.1.54 (registry+https://github.com/rust-lang/crates.io-index)", + "bitflags", ] [[package]] name = "regex" -version = "1.1.6" +version = "1.5.4" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d07a8629359eb56f1e2fb1652bb04212c072a87ba68546a04065d525673ac461" dependencies = [ - "aho-corasick 0.7.3 (registry+https://github.com/rust-lang/crates.io-index)", - "memchr 2.2.0 (registry+https://github.com/rust-lang/crates.io-index)", - "regex-syntax 0.6.6 (registry+https://github.com/rust-lang/crates.io-index)", - "thread_local 0.3.6 (registry+https://github.com/rust-lang/crates.io-index)", - "utf8-ranges 1.0.2 (registry+https://github.com/rust-lang/crates.io-index)", + "aho-corasick", + "memchr", + "regex-syntax", ] [[package]] name = "regex-syntax" -version = "0.6.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "ucd-util 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "remove_dir_all" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "winapi 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "reqwest" -version = "0.9.17" +version = "0.6.25" source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "base64 0.10.1 (registry+https://github.com/rust-lang/crates.io-index)", - "bytes 0.4.12 (registry+https://github.com/rust-lang/crates.io-index)", - "cookie 0.12.0 (registry+https://github.com/rust-lang/crates.io-index)", - "cookie_store 0.7.0 (registry+https://github.com/rust-lang/crates.io-index)", - "encoding_rs 0.8.17 (registry+https://github.com/rust-lang/crates.io-index)", - "flate2 1.0.7 (registry+https://github.com/rust-lang/crates.io-index)", - "futures 0.1.27 (registry+https://github.com/rust-lang/crates.io-index)", - "http 0.1.17 (registry+https://github.com/rust-lang/crates.io-index)", - "hyper 0.12.29 (registry+https://github.com/rust-lang/crates.io-index)", - "hyper-tls 0.3.2 (registry+https://github.com/rust-lang/crates.io-index)", - "log 0.4.6 (registry+https://github.com/rust-lang/crates.io-index)", - "mime 0.3.13 (registry+https://github.com/rust-lang/crates.io-index)", - "mime_guess 2.0.0-alpha.6 (registry+https://github.com/rust-lang/crates.io-index)", - "native-tls 0.2.3 (registry+https://github.com/rust-lang/crates.io-index)", - "serde 1.0.91 (registry+https://github.com/rust-lang/crates.io-index)", - "serde_json 1.0.39 (registry+https://github.com/rust-lang/crates.io-index)", - "serde_urlencoded 0.5.5 (registry+https://github.com/rust-lang/crates.io-index)", - "time 0.1.42 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio 0.1.20 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio-executor 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio-io 0.1.12 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio-threadpool 0.1.14 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio-timer 0.2.11 (registry+https://github.com/rust-lang/crates.io-index)", - "url 1.7.2 (registry+https://github.com/rust-lang/crates.io-index)", - "uuid 0.7.4 (registry+https://github.com/rust-lang/crates.io-index)", -] +checksum = "f497285884f3fcff424ffc933e56d7cbca511def0c9831a7f9b5f6153e3cc89b" [[package]] name = "ring" -version = "0.14.6" +version = "0.16.20" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3053cf52e236a3ed746dfc745aa9cacf1b791d846bdaf412f60a8d7d6e17c8fc" dependencies = [ - "cc 1.0.37 (registry+https://github.com/rust-lang/crates.io-index)", - "lazy_static 1.3.0 (registry+https://github.com/rust-lang/crates.io-index)", - "libc 0.2.55 (registry+https://github.com/rust-lang/crates.io-index)", - "spin 0.5.0 (registry+https://github.com/rust-lang/crates.io-index)", - "untrusted 0.6.2 (registry+https://github.com/rust-lang/crates.io-index)", - "winapi 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)", + "cc", + "libc", + "once_cell", + "spin", + "untrusted", + "web-sys", + "winapi", ] [[package]] -name = "rustc-demangle" -version = "0.1.14" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "rustc_version" -version = "0.2.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "semver 0.9.0 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "ryu" -version = "0.2.8" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "safemem" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "schannel" -version = "0.1.15" +name = "rustls" +version = "0.20.2" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d37e5e2290f3e040b594b1a9e04377c2c671f1a1cfd9bfdef82106ac1c113f84" dependencies = [ - "lazy_static 1.3.0 (registry+https://github.com/rust-lang/crates.io-index)", - "winapi 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)", + "log", + "ring", + "sct", + "webpki", ] [[package]] name = "scopeguard" -version = "0.3.3" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "security-framework" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "core-foundation 0.6.4 (registry+https://github.com/rust-lang/crates.io-index)", - "core-foundation-sys 0.6.2 (registry+https://github.com/rust-lang/crates.io-index)", - "libc 0.2.55 (registry+https://github.com/rust-lang/crates.io-index)", - "security-framework-sys 0.3.1 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "security-framework-sys" -version = "0.3.1" +version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "core-foundation-sys 0.6.2 (registry+https://github.com/rust-lang/crates.io-index)", -] +checksum = "d29ab0c6d3fc0ee92fe66e2d99f700eab17a8d57d1c1d3b748380fb20baa78cd" [[package]] -name = "semver" -version = "0.9.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "semver-parser 0.7.0 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "semver-parser" +name = "sct" version = "0.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "serde" -version = "1.0.91" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "serde_derive 1.0.91 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "serde_derive" -version = "1.0.91" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "proc-macro2 0.4.30 (registry+https://github.com/rust-lang/crates.io-index)", - "quote 0.6.12 (registry+https://github.com/rust-lang/crates.io-index)", - "syn 0.15.34 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "serde_json" -version = "1.0.39" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "itoa 0.4.4 (registry+https://github.com/rust-lang/crates.io-index)", - "ryu 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)", - "serde 1.0.91 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "serde_urlencoded" -version = "0.5.5" -source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d53dcdb7c9f8158937a7981b48accfd39a43af418591a5d008c7b22b5e1b7ca4" dependencies = [ - "dtoa 0.4.4 (registry+https://github.com/rust-lang/crates.io-index)", - "itoa 0.4.4 (registry+https://github.com/rust-lang/crates.io-index)", - "serde 1.0.91 (registry+https://github.com/rust-lang/crates.io-index)", - "url 1.7.2 (registry+https://github.com/rust-lang/crates.io-index)", + "ring", + "untrusted", ] [[package]] name = "sha2" -version = "0.7.1" +version = "0.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "900d964dd36bb15bcf2f2b35694c072feab74969a54f2bbeec7a2d725d2bdcb6" dependencies = [ - "block-buffer 0.3.3 (registry+https://github.com/rust-lang/crates.io-index)", - "byte-tools 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)", - "digest 0.7.6 (registry+https://github.com/rust-lang/crates.io-index)", - "fake-simd 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)", + "cfg-if", + "cpufeatures", + "digest", ] [[package]] name = "siphasher" -version = "0.2.3" +version = "0.3.7" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "533494a8f9b724d33625ab53c6c4800f7cc445895924a8ef649222dcb76e938b" [[package]] name = "slab" -version = "0.4.2" +version = "0.4.5" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9def91fd1e018fe007022791f865d0ccc9b3a0d5001e01aabb8b40e46000afb5" [[package]] name = "smallvec" -version = "0.6.9" +version = "1.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ecab6c735a6bb4139c0caafd0cc3635748bbb3acf4550e8138122099251f309" [[package]] name = "socket2" -version = "0.3.9" +version = "0.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5dc90fe6c7be1a323296982db1836d1ea9e47b6839496dde9a541bc496df3516" dependencies = [ - "cfg-if 0.1.9 (registry+https://github.com/rust-lang/crates.io-index)", - "libc 0.2.55 (registry+https://github.com/rust-lang/crates.io-index)", - "redox_syscall 0.1.54 (registry+https://github.com/rust-lang/crates.io-index)", - "winapi 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)", + "libc", + "winapi", ] [[package]] name = "spin" -version = "0.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "stable_deref_trait" -version = "1.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "string" -version = "0.1.3" +version = "0.5.2" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6e63cff320ae2c57904679ba7cb63280a3dc4613885beafb148ee7bf9aa9042d" [[package]] name = "stringprep" version = "0.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ee348cb74b87454fff4b551cbf727025810a004f88aeacae7f85b87f4e9a1c1" dependencies = [ - "unicode-bidi 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)", - "unicode-normalization 0.1.8 (registry+https://github.com/rust-lang/crates.io-index)", + "unicode-bidi", + "unicode-normalization", ] [[package]] name = "strsim" version = "0.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ea5119cdb4c55b55d432abb513a0429384878c15dde60cc77b1c99de1a95a6a" [[package]] -name = "syn" -version = "0.15.34" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "proc-macro2 0.4.30 (registry+https://github.com/rust-lang/crates.io-index)", - "quote 0.6.12 (registry+https://github.com/rust-lang/crates.io-index)", - "unicode-xid 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "synstructure" -version = "0.10.2" +name = "subtle" +version = "2.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "proc-macro2 0.4.30 (registry+https://github.com/rust-lang/crates.io-index)", - "quote 0.6.12 (registry+https://github.com/rust-lang/crates.io-index)", - "syn 0.15.34 (registry+https://github.com/rust-lang/crates.io-index)", - "unicode-xid 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)", -] +checksum = "6bdef32e8150c2a081110b42772ffe7d7c9032b606bc226c8260fd97e0976601" [[package]] -name = "tempfile" -version = "3.0.8" +name = "syn" +version = "1.0.82" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8daf5dd0bb60cbd4137b1b587d2fc0ae729bc07cf01cd70b36a1ed5ade3b9d59" dependencies = [ - "cfg-if 0.1.9 (registry+https://github.com/rust-lang/crates.io-index)", - "libc 0.2.55 (registry+https://github.com/rust-lang/crates.io-index)", - "rand 0.6.5 (registry+https://github.com/rust-lang/crates.io-index)", - "redox_syscall 0.1.54 (registry+https://github.com/rust-lang/crates.io-index)", - "remove_dir_all 0.5.1 (registry+https://github.com/rust-lang/crates.io-index)", - "winapi 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)", + "proc-macro2", + "quote", + "unicode-xid", ] [[package]] name = "termcolor" -version = "1.0.4" +version = "1.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2dfed899f0eb03f32ee8c6a0aabdb8a7949659e3466561fc0adf54e26d88c5f4" dependencies = [ - "wincolor 1.0.1 (registry+https://github.com/rust-lang/crates.io-index)", + "winapi-util", ] [[package]] -name = "termion" -version = "1.5.2" +name = "textwrap" +version = "0.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d326610f408c7a4eb6f51c37c330e496b08506c9457c9d34287ecc38809fb060" dependencies = [ - "libc 0.2.55 (registry+https://github.com/rust-lang/crates.io-index)", - "numtoa 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)", - "redox_syscall 0.1.54 (registry+https://github.com/rust-lang/crates.io-index)", - "redox_termios 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)", + "unicode-width", ] [[package]] -name = "textwrap" -version = "0.11.0" +name = "time" +version = "0.1.43" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ca8a50ef2360fbd1eeb0ecd46795a87a19024eb4b53c5dc916ca1fd95fe62438" dependencies = [ - "unicode-width 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)", + "libc", + "winapi", ] [[package]] -name = "thread_local" -version = "0.3.6" +name = "tinyvec" +version = "1.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2c1c1d5a42b6245520c249549ec267180beaffcc0615401ac8e31853d4b6d8d2" dependencies = [ - "lazy_static 1.3.0 (registry+https://github.com/rust-lang/crates.io-index)", + "tinyvec_macros", ] [[package]] -name = "time" -version = "0.1.42" +name = "tinyvec_macros" +version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "libc 0.2.55 (registry+https://github.com/rust-lang/crates.io-index)", - "redox_syscall 0.1.54 (registry+https://github.com/rust-lang/crates.io-index)", - "winapi 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)", -] +checksum = "cda74da7e1a664f795bb1f8a87ec406fb89a02522cf6e50620d016add6dbbf5c" [[package]] name = "tokio" -version = "0.1.20" +version = "1.14.0" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "70e992e41e0d2fb9f755b37446f20900f64446ef54874f40a60c78f021ac6144" dependencies = [ - "bytes 0.4.12 (registry+https://github.com/rust-lang/crates.io-index)", - "futures 0.1.27 (registry+https://github.com/rust-lang/crates.io-index)", - "mio 0.6.18 (registry+https://github.com/rust-lang/crates.io-index)", - "num_cpus 1.10.0 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio-current-thread 0.1.6 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio-executor 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio-io 0.1.12 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio-reactor 0.1.9 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio-tcp 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio-threadpool 0.1.14 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio-timer 0.2.11 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio-trace-core 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)", + "autocfg", + "bytes", + "libc", + "memchr", + "mio", + "pin-project-lite", + "winapi", ] [[package]] -name = "tokio-buf" -version = "0.1.1" +name = "tokio-postgres" +version = "0.7.5" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4b6c8b33df661b548dcd8f9bf87debb8c56c05657ed291122e1188698c2ece95" dependencies = [ - "bytes 0.4.12 (registry+https://github.com/rust-lang/crates.io-index)", - "either 1.5.2 (registry+https://github.com/rust-lang/crates.io-index)", - "futures 0.1.27 (registry+https://github.com/rust-lang/crates.io-index)", + "async-trait", + "byteorder", + "bytes", + "fallible-iterator", + "futures", + "log", + "parking_lot", + "percent-encoding", + "phf", + "pin-project-lite", + "postgres-protocol", + "postgres-types", + "socket2", + "tokio", + "tokio-util", ] [[package]] -name = "tokio-current-thread" -version = "0.1.6" +name = "tokio-util" +version = "0.6.9" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9e99e1983e5d376cd8eb4b66604d2e99e79f5bd988c3055891dcd8c9e2604cc0" dependencies = [ - "futures 0.1.27 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio-executor 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)", + "bytes", + "futures-core", + "futures-sink", + "log", + "pin-project-lite", + "tokio", ] [[package]] -name = "tokio-executor" -version = "0.1.7" +name = "typenum" +version = "1.14.0" source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "crossbeam-utils 0.6.5 (registry+https://github.com/rust-lang/crates.io-index)", - "futures 0.1.27 (registry+https://github.com/rust-lang/crates.io-index)", -] +checksum = "b63708a265f51345575b27fe43f9500ad611579e764c79edbc2037b1121959ec" [[package]] -name = "tokio-io" -version = "0.1.12" +name = "unicode-bidi" +version = "0.3.7" source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "bytes 0.4.12 (registry+https://github.com/rust-lang/crates.io-index)", - "futures 0.1.27 (registry+https://github.com/rust-lang/crates.io-index)", - "log 0.4.6 (registry+https://github.com/rust-lang/crates.io-index)", -] +checksum = "1a01404663e3db436ed2746d9fefef640d868edae3cceb81c3b8d5732fda678f" [[package]] -name = "tokio-reactor" -version = "0.1.9" +name = "unicode-normalization" +version = "0.1.19" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d54590932941a9e9266f0832deed84ebe1bf2e4c9e4a3554d393d18f5e854bf9" dependencies = [ - "crossbeam-utils 0.6.5 (registry+https://github.com/rust-lang/crates.io-index)", - "futures 0.1.27 (registry+https://github.com/rust-lang/crates.io-index)", - "lazy_static 1.3.0 (registry+https://github.com/rust-lang/crates.io-index)", - "log 0.4.6 (registry+https://github.com/rust-lang/crates.io-index)", - "mio 0.6.18 (registry+https://github.com/rust-lang/crates.io-index)", - "num_cpus 1.10.0 (registry+https://github.com/rust-lang/crates.io-index)", - "parking_lot 0.7.1 (registry+https://github.com/rust-lang/crates.io-index)", - "slab 0.4.2 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio-executor 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio-io 0.1.12 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio-sync 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)", + "tinyvec", ] [[package]] -name = "tokio-sync" -version = "0.1.5" +name = "unicode-width" +version = "0.1.9" source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "fnv 1.0.6 (registry+https://github.com/rust-lang/crates.io-index)", - "futures 0.1.27 (registry+https://github.com/rust-lang/crates.io-index)", -] +checksum = "3ed742d4ea2bd1176e236172c8429aaf54486e7ac098db29ffe6529e0ce50973" [[package]] -name = "tokio-tcp" -version = "0.1.3" +name = "unicode-xid" +version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "bytes 0.4.12 (registry+https://github.com/rust-lang/crates.io-index)", - "futures 0.1.27 (registry+https://github.com/rust-lang/crates.io-index)", - "iovec 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)", - "mio 0.6.18 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio-io 0.1.12 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio-reactor 0.1.9 (registry+https://github.com/rust-lang/crates.io-index)", -] +checksum = "8ccb82d61f80a663efe1f787a51b16b5a51e3314d6ac365b08639f52387b33f3" [[package]] -name = "tokio-threadpool" -version = "0.1.14" +name = "untrusted" +version = "0.7.1" source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "crossbeam-deque 0.7.1 (registry+https://github.com/rust-lang/crates.io-index)", - "crossbeam-queue 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)", - "crossbeam-utils 0.6.5 (registry+https://github.com/rust-lang/crates.io-index)", - "futures 0.1.27 (registry+https://github.com/rust-lang/crates.io-index)", - "log 0.4.6 (registry+https://github.com/rust-lang/crates.io-index)", - "num_cpus 1.10.0 (registry+https://github.com/rust-lang/crates.io-index)", - "rand 0.6.5 (registry+https://github.com/rust-lang/crates.io-index)", - "slab 0.4.2 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio-executor 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)", -] +checksum = "a156c684c91ea7d62626509bce3cb4e1d9ed5c4d978f7b4352658f96a4c26b4a" [[package]] -name = "tokio-timer" -version = "0.2.11" +name = "ureq" +version = "2.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c5c448dcb78ec38c7d59ec61f87f70a98ea19171e06c139357e012ee226fec90" dependencies = [ - "crossbeam-utils 0.6.5 (registry+https://github.com/rust-lang/crates.io-index)", - "futures 0.1.27 (registry+https://github.com/rust-lang/crates.io-index)", - "slab 0.4.2 (registry+https://github.com/rust-lang/crates.io-index)", - "tokio-executor 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)", + "base64", + "chunked_transfer", + "log", + "once_cell", + "rustls", + "url", + "webpki", + "webpki-roots", ] [[package]] -name = "tokio-trace-core" -version = "0.1.0" +name = "url" +version = "2.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a507c383b2d33b5fc35d1861e77e6b383d158b2da5e14fe51b83dfedf6fd578c" dependencies = [ - "lazy_static 1.3.0 (registry+https://github.com/rust-lang/crates.io-index)", + "form_urlencoded", + "idna", + "matches", + "percent-encoding", ] [[package]] -name = "try-lock" -version = "0.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "try_from" -version = "0.3.2" +name = "vec_map" +version = "0.8.2" source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "cfg-if 0.1.9 (registry+https://github.com/rust-lang/crates.io-index)", -] +checksum = "f1bddf1187be692e79c5ffeab891132dfb0f236ed36a43c7ed39f1165ee20191" [[package]] -name = "typenum" -version = "1.10.0" +name = "version_check" +version = "0.9.3" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5fecdca9a5291cc2b8dcf7dc02453fee791a280f3743cb0905f8822ae463b3fe" [[package]] -name = "ucd-util" -version = "0.1.3" +name = "wasi" +version = "0.10.2+wasi-snapshot-preview1" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fd6fbd9a79829dd1ad0cc20627bf1ed606756a7f77edff7b66b7064f9cb327c6" [[package]] -name = "unicase" -version = "1.4.2" +name = "wasm-bindgen" +version = "0.2.78" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "632f73e236b219150ea279196e54e610f5dbafa5d61786303d4da54f84e47fce" dependencies = [ - "version_check 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)", + "cfg-if", + "wasm-bindgen-macro", ] [[package]] -name = "unicase" -version = "2.4.0" +name = "wasm-bindgen-backend" +version = "0.2.78" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a317bf8f9fba2476b4b2c85ef4c4af8ff39c3c7f0cdfeed4f82c34a880aa837b" dependencies = [ - "version_check 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)", + "bumpalo", + "lazy_static", + "log", + "proc-macro2", + "quote", + "syn", + "wasm-bindgen-shared", ] [[package]] -name = "unicode-bidi" -version = "0.3.4" +name = "wasm-bindgen-macro" +version = "0.2.78" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d56146e7c495528bf6587663bea13a8eb588d39b36b679d83972e1a2dbbdacf9" dependencies = [ - "matches 0.1.8 (registry+https://github.com/rust-lang/crates.io-index)", + "quote", + "wasm-bindgen-macro-support", ] [[package]] -name = "unicode-normalization" -version = "0.1.8" +name = "wasm-bindgen-macro-support" +version = "0.2.78" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7803e0eea25835f8abdc585cd3021b3deb11543c6fe226dcd30b228857c5c5ab" dependencies = [ - "smallvec 0.6.9 (registry+https://github.com/rust-lang/crates.io-index)", + "proc-macro2", + "quote", + "syn", + "wasm-bindgen-backend", + "wasm-bindgen-shared", ] [[package]] -name = "unicode-width" -version = "0.1.5" +name = "wasm-bindgen-shared" +version = "0.2.78" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0237232789cf037d5480773fe568aac745bfe2afbc11a863e97901780a6b47cc" [[package]] -name = "unicode-xid" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "untrusted" -version = "0.6.2" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "url" -version = "1.7.2" +name = "web-sys" +version = "0.3.55" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "38eb105f1c59d9eaa6b5cdc92b859d85b926e82cb2e0945cd0c9259faa6fe9fb" dependencies = [ - "idna 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)", - "matches 0.1.8 (registry+https://github.com/rust-lang/crates.io-index)", - "percent-encoding 1.0.1 (registry+https://github.com/rust-lang/crates.io-index)", + "js-sys", + "wasm-bindgen", ] [[package]] -name = "utf8-ranges" -version = "1.0.2" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "uuid" -version = "0.7.4" +name = "webpki" +version = "0.22.0" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f095d78192e208183081cc07bc5515ef55216397af48b873e5edcd72637fa1bd" dependencies = [ - "rand 0.6.5 (registry+https://github.com/rust-lang/crates.io-index)", + "ring", + "untrusted", ] [[package]] -name = "vcpkg" -version = "0.2.6" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "vec_map" -version = "0.8.1" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "version_check" -version = "0.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "want" -version = "0.0.6" +name = "webpki-roots" +version = "0.22.1" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c475786c6f47219345717a043a37ec04cb4bc185e28853adcc4fa0a947eba630" dependencies = [ - "futures 0.1.27 (registry+https://github.com/rust-lang/crates.io-index)", - "log 0.4.6 (registry+https://github.com/rust-lang/crates.io-index)", - "try-lock 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)", + "webpki", ] [[package]] name = "winapi" -version = "0.2.8" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "winapi" -version = "0.3.7" +version = "0.3.9" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419" dependencies = [ - "winapi-i686-pc-windows-gnu 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)", - "winapi-x86_64-pc-windows-gnu 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)", + "winapi-i686-pc-windows-gnu", + "winapi-x86_64-pc-windows-gnu", ] [[package]] -name = "winapi-build" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] name = "winapi-i686-pc-windows-gnu" version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6" [[package]] name = "winapi-util" -version = "0.1.2" +version = "0.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "70ec6ce85bb158151cae5e5c87f95a8e97d2c0c4b001223f33a334e3ce5de178" dependencies = [ - "winapi 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)", + "winapi", ] [[package]] name = "winapi-x86_64-pc-windows-gnu" version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "wincolor" -version = "1.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "winapi 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)", - "winapi-util 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "ws2_32-sys" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)", - "winapi-build 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[metadata] -"checksum adler32 1.0.3 (registry+https://github.com/rust-lang/crates.io-index)" = "7e522997b529f05601e05166c07ed17789691f562762c7f3b987263d2dedee5c" -"checksum aho-corasick 0.7.3 (registry+https://github.com/rust-lang/crates.io-index)" = "e6f484ae0c99fec2e858eb6134949117399f222608d84cadb3f58c1f97c2364c" -"checksum ansi_term 0.11.0 (registry+https://github.com/rust-lang/crates.io-index)" = "ee49baf6cb617b853aa8d93bf420db2383fab46d314482ca2803b40d5fde979b" -"checksum arrayref 0.3.5 (registry+https://github.com/rust-lang/crates.io-index)" = "0d382e583f07208808f6b1249e60848879ba3543f57c32277bf52d69c2f0f0ee" -"checksum arrayvec 0.4.10 (registry+https://github.com/rust-lang/crates.io-index)" = "92c7fb76bc8826a8b33b4ee5bb07a247a81e76764ab4d55e8f73e3a4d8808c71" -"checksum atty 0.2.11 (registry+https://github.com/rust-lang/crates.io-index)" = "9a7d5b8723950951411ee34d271d99dddcc2035a16ab25310ea2c8cfd4369652" -"checksum autocfg 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)" = "0e49efa51329a5fd37e7c79db4621af617cd4e3e5bc224939808d076077077bf" -"checksum backtrace 0.3.26 (registry+https://github.com/rust-lang/crates.io-index)" = "1a13fc43f04daf08ab4f71e3d27e1fc27fc437d3e95ac0063a796d92fb40f39b" -"checksum backtrace-sys 0.1.28 (registry+https://github.com/rust-lang/crates.io-index)" = "797c830ac25ccc92a7f8a7b9862bde440715531514594a6154e3d4a54dd769b6" -"checksum base64 0.10.1 (registry+https://github.com/rust-lang/crates.io-index)" = "0b25d992356d2eb0ed82172f5248873db5560c4721f564b13cb5193bda5e668e" -"checksum base64 0.6.0 (registry+https://github.com/rust-lang/crates.io-index)" = "96434f987501f0ed4eb336a411e0631ecd1afa11574fe148587adc4ff96143c9" -"checksum bitflags 1.0.4 (registry+https://github.com/rust-lang/crates.io-index)" = "228047a76f468627ca71776ecdebd732a3423081fcf5125585bcd7c49886ce12" -"checksum block-buffer 0.3.3 (registry+https://github.com/rust-lang/crates.io-index)" = "a076c298b9ecdb530ed9d967e74a6027d6a7478924520acddcddc24c1c8ab3ab" -"checksum build_const 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "39092a32794787acd8525ee150305ff051b0aa6cc2abaf193924f5ab05425f39" -"checksum byte-tools 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)" = "560c32574a12a89ecd91f5e742165893f86e3ab98d21f8ea548658eb9eef5f40" -"checksum byteorder 1.3.1 (registry+https://github.com/rust-lang/crates.io-index)" = "a019b10a2a7cdeb292db131fc8113e57ea2a908f6e7894b0c3c671893b65dbeb" -"checksum bytes 0.4.12 (registry+https://github.com/rust-lang/crates.io-index)" = "206fdffcfa2df7cbe15601ef46c813fce0965eb3286db6b56c583b814b51c81c" -"checksum cc 1.0.37 (registry+https://github.com/rust-lang/crates.io-index)" = "39f75544d7bbaf57560d2168f28fd649ff9c76153874db88bdbdfd839b1a7e7d" -"checksum cfg-if 0.1.9 (registry+https://github.com/rust-lang/crates.io-index)" = "b486ce3ccf7ffd79fdeb678eac06a9e6c09fc88d33836340becb8fffe87c5e33" -"checksum chrono 0.4.6 (registry+https://github.com/rust-lang/crates.io-index)" = "45912881121cb26fad7c38c17ba7daa18764771836b34fab7d3fbd93ed633878" -"checksum clap 2.33.0 (registry+https://github.com/rust-lang/crates.io-index)" = "5067f5bb2d80ef5d68b4c87db81601f0b75bca627bc2ef76b141d7b846a3c6d9" -"checksum cloudabi 0.0.3 (registry+https://github.com/rust-lang/crates.io-index)" = "ddfc5b9aa5d4507acaf872de71051dfd0e309860e88966e1051e462a077aac4f" -"checksum constant_time_eq 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)" = "8ff012e225ce166d4422e0e78419d901719760f62ae2b7969ca6b564d1b54a9e" -"checksum cookie 0.12.0 (registry+https://github.com/rust-lang/crates.io-index)" = "888604f00b3db336d2af898ec3c1d5d0ddf5e6d462220f2ededc33a87ac4bbd5" -"checksum cookie_store 0.7.0 (registry+https://github.com/rust-lang/crates.io-index)" = "46750b3f362965f197996c4448e4a0935e791bf7d6631bfce9ee0af3d24c919c" -"checksum core-foundation 0.6.4 (registry+https://github.com/rust-lang/crates.io-index)" = "25b9e03f145fd4f2bf705e07b900cd41fc636598fe5dc452fd0db1441c3f496d" -"checksum core-foundation-sys 0.6.2 (registry+https://github.com/rust-lang/crates.io-index)" = "e7ca8a5221364ef15ce201e8ed2f609fc312682a8f4e0e3d4aa5879764e0fa3b" -"checksum crc 1.8.1 (registry+https://github.com/rust-lang/crates.io-index)" = "d663548de7f5cca343f1e0a48d14dcfb0e9eb4e079ec58883b7251539fa10aeb" -"checksum crc32fast 1.2.0 (registry+https://github.com/rust-lang/crates.io-index)" = "ba125de2af0df55319f41944744ad91c71113bf74a4646efff39afe1f6842db1" -"checksum crossbeam-deque 0.7.1 (registry+https://github.com/rust-lang/crates.io-index)" = "b18cd2e169ad86297e6bc0ad9aa679aee9daa4f19e8163860faf7c164e4f5a71" -"checksum crossbeam-epoch 0.7.1 (registry+https://github.com/rust-lang/crates.io-index)" = "04c9e3102cc2d69cd681412141b390abd55a362afc1540965dad0ad4d34280b4" -"checksum crossbeam-queue 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)" = "7c979cd6cfe72335896575c6b5688da489e420d36a27a0b9eb0c73db574b4a4b" -"checksum crossbeam-utils 0.6.5 (registry+https://github.com/rust-lang/crates.io-index)" = "f8306fcef4a7b563b76b7dd949ca48f52bc1141aa067d2ea09565f3e2652aa5c" -"checksum crypto-mac 0.5.2 (registry+https://github.com/rust-lang/crates.io-index)" = "0999b4ff4d3446d4ddb19a63e9e00c1876e75cd7000d20e57a693b4b3f08d958" -"checksum derive_more 0.14.0 (registry+https://github.com/rust-lang/crates.io-index)" = "fbe9f11be34f800b3ecaaed0ec9ec2e015d1d0ba0c8644c1310f73d6e8994615" -"checksum digest 0.7.6 (registry+https://github.com/rust-lang/crates.io-index)" = "03b072242a8cbaf9c145665af9d250c59af3b958f83ed6824e13533cf76d5b90" -"checksum dtoa 0.4.4 (registry+https://github.com/rust-lang/crates.io-index)" = "ea57b42383d091c85abcc2706240b94ab2a8fa1fc81c10ff23c4de06e2a90b5e" -"checksum either 1.5.2 (registry+https://github.com/rust-lang/crates.io-index)" = "5527cfe0d098f36e3f8839852688e63c8fff1c90b2b405aef730615f9a7bcf7b" -"checksum encoding 0.3.0-dev (git+https://github.com/lifthrasiir/rust-encoding)" = "<none>" -"checksum encoding-index-japanese 1.20141219.6 (git+https://github.com/lifthrasiir/rust-encoding)" = "<none>" -"checksum encoding-index-korean 1.20141219.6 (git+https://github.com/lifthrasiir/rust-encoding)" = "<none>" -"checksum encoding-index-simpchinese 1.20160120.0 (git+https://github.com/lifthrasiir/rust-encoding)" = "<none>" -"checksum encoding-index-singlebyte 1.20160120.0 (git+https://github.com/lifthrasiir/rust-encoding)" = "<none>" -"checksum encoding-index-tradchinese 1.20141219.6 (git+https://github.com/lifthrasiir/rust-encoding)" = "<none>" -"checksum encoding-types 0.2.0 (git+https://github.com/lifthrasiir/rust-encoding)" = "<none>" -"checksum encoding_index_tests 0.1.5 (git+https://github.com/lifthrasiir/rust-encoding)" = "<none>" -"checksum encoding_rs 0.8.17 (registry+https://github.com/rust-lang/crates.io-index)" = "4155785c79f2f6701f185eb2e6b4caf0555ec03477cb4c70db67b465311620ed" -"checksum env_logger 0.6.1 (registry+https://github.com/rust-lang/crates.io-index)" = "b61fa891024a945da30a9581546e8cfaf5602c7b3f4c137a2805cf388f92075a" -"checksum error-chain 0.12.1 (registry+https://github.com/rust-lang/crates.io-index)" = "3ab49e9dcb602294bc42f9a7dfc9bc6e936fca4418ea300dbfb84fe16de0b7d9" -"checksum failure 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)" = "795bd83d3abeb9220f257e597aa0080a508b27533824adf336529648f6abf7e2" -"checksum failure_derive 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)" = "ea1063915fd7ef4309e222a5a07cf9c319fb9c7836b1f89b85458672dbb127e1" -"checksum fake-simd 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)" = "e88a8acf291dafb59c2d96e8f59828f3838bb1a70398823ade51a84de6a6deed" -"checksum fallible-iterator 0.1.6 (registry+https://github.com/rust-lang/crates.io-index)" = "eb7217124812dc5672b7476d0c2d20cfe9f7c0f1ba0904b674a9762a0212f72e" -"checksum flate2 1.0.7 (registry+https://github.com/rust-lang/crates.io-index)" = "f87e68aa82b2de08a6e037f1385455759df6e445a8df5e005b4297191dbf18aa" -"checksum fnv 1.0.6 (registry+https://github.com/rust-lang/crates.io-index)" = "2fad85553e09a6f881f739c29f0b00b0f01357c743266d478b68951ce23285f3" -"checksum foreign-types 0.3.2 (registry+https://github.com/rust-lang/crates.io-index)" = "f6f339eb8adc052cd2ca78910fda869aefa38d22d5cb648e6485e4d3fc06f3b1" -"checksum foreign-types-shared 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "00b0228411908ca8685dba7fc2cdd70ec9990a6e753e89b6ac91a84c40fbaf4b" -"checksum fuchsia-cprng 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "a06f77d526c1a601b7c4cdd98f54b5eaabffc14d5f2f0296febdc7f357c6d3ba" -"checksum fuchsia-zircon 0.3.3 (registry+https://github.com/rust-lang/crates.io-index)" = "2e9763c69ebaae630ba35f74888db465e49e259ba1bc0eda7d06f4a067615d82" -"checksum fuchsia-zircon-sys 0.3.3 (registry+https://github.com/rust-lang/crates.io-index)" = "3dcaa9ae7725d12cdb85b3ad99a434db70b468c09ded17e012d86b5c1010f7a7" -"checksum futures 0.1.27 (registry+https://github.com/rust-lang/crates.io-index)" = "a2037ec1c6c1c4f79557762eab1f7eae1f64f6cb418ace90fae88f0942b60139" -"checksum futures-cpupool 0.1.8 (registry+https://github.com/rust-lang/crates.io-index)" = "ab90cde24b3319636588d0c35fe03b1333857621051837ed769faefb4c2162e4" -"checksum generic-array 0.9.0 (registry+https://github.com/rust-lang/crates.io-index)" = "ef25c5683767570c2bbd7deba372926a55eaae9982d7726ee2a1050239d45b9d" -"checksum h2 0.1.20 (registry+https://github.com/rust-lang/crates.io-index)" = "2b53def7bb0253af7718036fe9338c15defd209136819464384f3a553e07481b" -"checksum hex 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)" = "d6a22814455d41612f41161581c2883c0c6a1c41852729b17d5ed88f01e153aa" -"checksum hmac 0.5.0 (registry+https://github.com/rust-lang/crates.io-index)" = "44f3bdb08579d99d7dc761c0e266f13b5f2ab8c8c703b9fc9ef333cd8f48f55e" -"checksum http 0.1.17 (registry+https://github.com/rust-lang/crates.io-index)" = "eed324f0f0daf6ec10c474f150505af2c143f251722bf9dbd1261bd1f2ee2c1a" -"checksum http-body 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "6741c859c1b2463a423a1dbce98d418e6c3c3fc720fb0d45528657320920292d" -"checksum httparse 1.3.3 (registry+https://github.com/rust-lang/crates.io-index)" = "e8734b0cfd3bc3e101ec59100e101c2eecd19282202e87808b3037b442777a83" -"checksum humantime 1.2.0 (registry+https://github.com/rust-lang/crates.io-index)" = "3ca7e5f2e110db35f93b837c81797f3714500b81d517bf20c431b16d3ca4f114" -"checksum hyper 0.12.29 (registry+https://github.com/rust-lang/crates.io-index)" = "e2cd6adf83b3347d36e271f030621a8cf95fd1fd0760546b9fc5a24a0f1447c7" -"checksum hyper-tls 0.3.2 (registry+https://github.com/rust-lang/crates.io-index)" = "3a800d6aa50af4b5850b2b0f659625ce9504df908e9733b635720483be26174f" -"checksum idna 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)" = "38f09e0f0b1fb55fdee1f17470ad800da77af5186a1a76c026b679358b7e844e" -"checksum indexmap 1.0.2 (registry+https://github.com/rust-lang/crates.io-index)" = "7e81a7c05f79578dbc15793d8b619db9ba32b4577003ef3af1a91c416798c58d" -"checksum iovec 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)" = "dbe6e417e7d0975db6512b90796e8ce223145ac4e33c377e4a42882a0e88bb08" -"checksum itoa 0.4.4 (registry+https://github.com/rust-lang/crates.io-index)" = "501266b7edd0174f8530248f87f99c88fbe60ca4ef3dd486835b8d8d53136f7f" -"checksum kernel32-sys 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)" = "7507624b29483431c0ba2d82aece8ca6cdba9382bff4ddd0f7490560c056098d" -"checksum lazy_static 1.3.0 (registry+https://github.com/rust-lang/crates.io-index)" = "bc5729f27f159ddd61f4df6228e827e86643d4d3e7c32183cb30a1c08f604a14" -"checksum libarchive3-sys 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)" = "3cd3beae8f59a4c7a806523269b5392037577c150446e88d684dfa6de6031ca7" -"checksum libc 0.2.55 (registry+https://github.com/rust-lang/crates.io-index)" = "42914d39aad277d9e176efbdad68acb1d5443ab65afe0e0e4f0d49352a950880" -"checksum lock_api 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)" = "62ebf1391f6acad60e5c8b43706dde4582df75c06698ab44511d15016bc2442c" -"checksum log 0.4.6 (registry+https://github.com/rust-lang/crates.io-index)" = "c84ec4b527950aa83a329754b01dbe3f58361d1c5efacd1f6d68c494d08a17c6" -"checksum matches 0.1.8 (registry+https://github.com/rust-lang/crates.io-index)" = "7ffc5c5338469d4d3ea17d269fa8ea3512ad247247c30bd2df69e68309ed0a08" -"checksum md5 0.3.8 (registry+https://github.com/rust-lang/crates.io-index)" = "79c56d6a0b07f9e19282511c83fc5b086364cbae4ba8c7d5f190c3d9b0425a48" -"checksum memchr 1.0.2 (registry+https://github.com/rust-lang/crates.io-index)" = "148fab2e51b4f1cfc66da2a7c32981d1d3c083a803978268bb11fe4b86925e7a" -"checksum memchr 2.2.0 (registry+https://github.com/rust-lang/crates.io-index)" = "2efc7bc57c883d4a4d6e3246905283d8dae951bb3bd32f49d6ef297f546e1c39" -"checksum memoffset 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "0f9dc261e2b62d7a622bf416ea3c5245cdd5d9a7fcc428c0d06804dfce1775b3" -"checksum mime 0.3.13 (registry+https://github.com/rust-lang/crates.io-index)" = "3e27ca21f40a310bd06d9031785f4801710d566c184a6e15bad4f1d9b65f9425" -"checksum mime_guess 2.0.0-alpha.6 (registry+https://github.com/rust-lang/crates.io-index)" = "30de2e4613efcba1ec63d8133f344076952090c122992a903359be5a4f99c3ed" -"checksum miniz_oxide 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "c468f2369f07d651a5d0bb2c9079f8488a66d5466efe42d0c5c6466edcb7f71e" -"checksum miniz_oxide_c_api 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "b7fe927a42e3807ef71defb191dc87d4e24479b221e67015fe38ae2b7b447bab" -"checksum mio 0.6.18 (registry+https://github.com/rust-lang/crates.io-index)" = "9fbe95ae9216d99c944a1afa429fef2a2ed012b65b0840de5047a86a82969502" -"checksum miow 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "8c1f2f3b1cf331de6896aabf6e9d55dca90356cc9960cca7eaaf408a355ae919" -"checksum native-tls 0.2.3 (registry+https://github.com/rust-lang/crates.io-index)" = "4b2df1a4c22fd44a62147fd8f13dd0f95c9d8ca7b2610299b2a2f9cf8964274e" -"checksum net2 0.2.33 (registry+https://github.com/rust-lang/crates.io-index)" = "42550d9fb7b6684a6d404d9fa7250c2eb2646df731d1c06afc06dcee9e1bcf88" -"checksum nodrop 0.1.13 (registry+https://github.com/rust-lang/crates.io-index)" = "2f9667ddcc6cc8a43afc9b7917599d7216aa09c463919ea32c59ed6cac8bc945" -"checksum num-integer 0.1.41 (registry+https://github.com/rust-lang/crates.io-index)" = "b85e541ef8255f6cf42bbfe4ef361305c6c135d10919ecc26126c4e5ae94bc09" -"checksum num-traits 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)" = "6ba9a427cfca2be13aa6f6403b0b7e7368fe982bfa16fccc450ce74c46cd9b32" -"checksum num_cpus 1.10.0 (registry+https://github.com/rust-lang/crates.io-index)" = "1a23f0ed30a54abaa0c7e83b1d2d87ada7c3c23078d1d87815af3e3b6385fbba" -"checksum numtoa 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "b8f8bdf33df195859076e54ab11ee78a1b208382d3a26ec40d142ffc1ecc49ef" -"checksum openssl 0.10.23 (registry+https://github.com/rust-lang/crates.io-index)" = "97c140cbb82f3b3468193dd14c1b88def39f341f68257f8a7fe8ed9ed3f628a5" -"checksum openssl-probe 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)" = "77af24da69f9d9341038eba93a073b1fdaaa1b788221b00a69bce9e762cb32de" -"checksum openssl-sys 0.9.47 (registry+https://github.com/rust-lang/crates.io-index)" = "75bdd6dbbb4958d38e47a1d2348847ad1eb4dc205dc5d37473ae504391865acc" -"checksum owning_ref 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)" = "49a4b8ea2179e6a2e27411d3bca09ca6dd630821cf6894c6c7c8467a8ee7ef13" -"checksum parking_lot 0.7.1 (registry+https://github.com/rust-lang/crates.io-index)" = "ab41b4aed082705d1056416ae4468b6ea99d52599ecf3169b00088d43113e337" -"checksum parking_lot_core 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)" = "94c8c7923936b28d546dfd14d4472eaf34c99b14e1c973a32b3e6d4eb04298c9" -"checksum percent-encoding 1.0.1 (registry+https://github.com/rust-lang/crates.io-index)" = "31010dd2e1ac33d5b46a5b413495239882813e0369f8ed8a5e266f173602f831" -"checksum phf 0.7.24 (registry+https://github.com/rust-lang/crates.io-index)" = "b3da44b85f8e8dfaec21adae67f95d93244b2ecf6ad2a692320598dcc8e6dd18" -"checksum phf_codegen 0.7.24 (registry+https://github.com/rust-lang/crates.io-index)" = "b03e85129e324ad4166b06b2c7491ae27fe3ec353af72e72cd1654c7225d517e" -"checksum phf_generator 0.7.24 (registry+https://github.com/rust-lang/crates.io-index)" = "09364cc93c159b8b06b1f4dd8a4398984503483891b0c26b867cf431fb132662" -"checksum phf_shared 0.7.24 (registry+https://github.com/rust-lang/crates.io-index)" = "234f71a15de2288bcb7e3b6515828d22af7ec8598ee6d24c3b526fa0a80b67a0" -"checksum pkg-config 0.3.14 (registry+https://github.com/rust-lang/crates.io-index)" = "676e8eb2b1b4c9043511a9b7bea0915320d7e502b0a079fb03f9635a5252b18c" -"checksum postgres 0.15.2 (registry+https://github.com/rust-lang/crates.io-index)" = "115dde90ef51af573580c035857badbece2aa5cde3de1dfb3c932969ca92a6c5" -"checksum postgres-protocol 0.3.2 (registry+https://github.com/rust-lang/crates.io-index)" = "2487e66455bf88a1b247bf08a3ce7fe5197ac6d67228d920b0ee6a0e97fd7312" -"checksum postgres-shared 0.4.2 (registry+https://github.com/rust-lang/crates.io-index)" = "ffac35b3e0029b404c24a3b82149b4e904f293e8ca4a327eefa24d3ca50df36f" -"checksum proc-macro2 0.4.30 (registry+https://github.com/rust-lang/crates.io-index)" = "cf3d2011ab5c909338f7887f4fc896d35932e29146c12c8d01da6b22a80ba759" -"checksum publicsuffix 1.5.2 (registry+https://github.com/rust-lang/crates.io-index)" = "5afecba86dcf1e4fd610246f89899d1924fe12e1e89f555eb7c7f710f3c5ad1d" -"checksum quick-error 1.2.2 (registry+https://github.com/rust-lang/crates.io-index)" = "9274b940887ce9addde99c4eee6b5c44cc494b182b97e73dc8ffdcb3397fd3f0" -"checksum quick-xml 0.14.0 (registry+https://github.com/rust-lang/crates.io-index)" = "0a8b2062cd4735d683121dbd525f5961226936229b0ac6bbbc40b34155744a41" -"checksum quote 0.6.12 (registry+https://github.com/rust-lang/crates.io-index)" = "faf4799c5d274f3868a4aae320a0a182cbd2baee377b378f080e16a23e9d80db" -"checksum rand 0.3.23 (registry+https://github.com/rust-lang/crates.io-index)" = "64ac302d8f83c0c1974bf758f6b041c6c8ada916fbb44a609158ca8b064cc76c" -"checksum rand 0.4.6 (registry+https://github.com/rust-lang/crates.io-index)" = "552840b97013b1a26992c11eac34bdd778e464601a4c2054b5f0bff7c6761293" -"checksum rand 0.6.5 (registry+https://github.com/rust-lang/crates.io-index)" = "6d71dacdc3c88c1fde3885a3be3fbab9f35724e6ce99467f7d9c5026132184ca" -"checksum rand_chacha 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "556d3a1ca6600bfcbab7c7c91ccb085ac7fbbcd70e008a98742e7847f4f7bcef" -"checksum rand_core 0.3.1 (registry+https://github.com/rust-lang/crates.io-index)" = "7a6fdeb83b075e8266dcc8762c22776f6877a63111121f5f8c7411e5be7eed4b" -"checksum rand_core 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)" = "d0e7a549d590831370895ab7ba4ea0c1b6b011d106b5ff2da6eee112615e6dc0" -"checksum rand_hc 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "7b40677c7be09ae76218dc623efbf7b18e34bced3f38883af07bb75630a21bc4" -"checksum rand_isaac 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "ded997c9d5f13925be2a6fd7e66bf1872597f759fd9dd93513dd7e92e5a5ee08" -"checksum rand_jitter 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)" = "1166d5c91dc97b88d1decc3285bb0a99ed84b05cfd0bc2341bdf2d43fc41e39b" -"checksum rand_os 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)" = "7b75f676a1e053fc562eafbb47838d67c84801e38fc1ba459e8f180deabd5071" -"checksum rand_pcg 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)" = "abf9b09b01790cfe0364f52bf32995ea3c39f4d2dd011eac241d2914146d0b44" -"checksum rand_xorshift 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "cbf7e9e623549b0e21f6e97cf8ecf247c1a8fd2e8a992ae265314300b2455d5c" -"checksum rdrand 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)" = "678054eb77286b51581ba43620cc911abf02758c91f93f479767aed0f90458b2" -"checksum redox_syscall 0.1.54 (registry+https://github.com/rust-lang/crates.io-index)" = "12229c14a0f65c4f1cb046a3b52047cdd9da1f4b30f8a39c5063c8bae515e252" -"checksum redox_termios 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "7e891cfe48e9100a70a3b6eb652fef28920c117d366339687bd5576160db0f76" -"checksum regex 1.1.6 (registry+https://github.com/rust-lang/crates.io-index)" = "8f0a0bcab2fd7d1d7c54fa9eae6f43eddeb9ce2e7352f8518a814a4f65d60c58" -"checksum regex-syntax 0.6.6 (registry+https://github.com/rust-lang/crates.io-index)" = "dcfd8681eebe297b81d98498869d4aae052137651ad7b96822f09ceb690d0a96" -"checksum remove_dir_all 0.5.1 (registry+https://github.com/rust-lang/crates.io-index)" = "3488ba1b9a2084d38645c4c08276a1752dcbf2c7130d74f1569681ad5d2799c5" -"checksum reqwest 0.9.17 (registry+https://github.com/rust-lang/crates.io-index)" = "e57803405f8ea0eb041c1567dac36127e0c8caa1251c843cb03d43fd767b3d50" -"checksum ring 0.14.6 (registry+https://github.com/rust-lang/crates.io-index)" = "426bc186e3e95cac1e4a4be125a4aca7e84c2d616ffc02244eef36e2a60a093c" -"checksum rustc-demangle 0.1.14 (registry+https://github.com/rust-lang/crates.io-index)" = "ccc78bfd5acd7bf3e89cffcf899e5cb1a52d6fafa8dec2739ad70c9577a57288" -"checksum rustc_version 0.2.3 (registry+https://github.com/rust-lang/crates.io-index)" = "138e3e0acb6c9fb258b19b67cb8abd63c00679d2851805ea151465464fe9030a" -"checksum ryu 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)" = "b96a9549dc8d48f2c283938303c4b5a77aa29bfbc5b54b084fb1630408899a8f" -"checksum safemem 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)" = "e27a8b19b835f7aea908818e871f5cc3a5a186550c30773be987e155e8163d8f" -"checksum schannel 0.1.15 (registry+https://github.com/rust-lang/crates.io-index)" = "f2f6abf258d99c3c1c5c2131d99d064e94b7b3dd5f416483057f308fea253339" -"checksum scopeguard 0.3.3 (registry+https://github.com/rust-lang/crates.io-index)" = "94258f53601af11e6a49f722422f6e3425c52b06245a5cf9bc09908b174f5e27" -"checksum security-framework 0.3.1 (registry+https://github.com/rust-lang/crates.io-index)" = "eee63d0f4a9ec776eeb30e220f0bc1e092c3ad744b2a379e3993070364d3adc2" -"checksum security-framework-sys 0.3.1 (registry+https://github.com/rust-lang/crates.io-index)" = "9636f8989cbf61385ae4824b98c1aaa54c994d7d8b41f11c601ed799f0549a56" -"checksum semver 0.9.0 (registry+https://github.com/rust-lang/crates.io-index)" = "1d7eb9ef2c18661902cc47e535f9bc51b78acd254da71d375c2f6720d9a40403" -"checksum semver-parser 0.7.0 (registry+https://github.com/rust-lang/crates.io-index)" = "388a1df253eca08550bef6c72392cfe7c30914bf41df5269b68cbd6ff8f570a3" -"checksum serde 1.0.91 (registry+https://github.com/rust-lang/crates.io-index)" = "a72e9b96fa45ce22a4bc23da3858dfccfd60acd28a25bcd328a98fdd6bea43fd" -"checksum serde_derive 1.0.91 (registry+https://github.com/rust-lang/crates.io-index)" = "101b495b109a3e3ca8c4cbe44cf62391527cdfb6ba15821c5ce80bcd5ea23f9f" -"checksum serde_json 1.0.39 (registry+https://github.com/rust-lang/crates.io-index)" = "5a23aa71d4a4d43fdbfaac00eff68ba8a06a51759a89ac3304323e800c4dd40d" -"checksum serde_urlencoded 0.5.5 (registry+https://github.com/rust-lang/crates.io-index)" = "642dd69105886af2efd227f75a520ec9b44a820d65bc133a9131f7d229fd165a" -"checksum sha2 0.7.1 (registry+https://github.com/rust-lang/crates.io-index)" = "9eb6be24e4c23a84d7184280d2722f7f2731fcdd4a9d886efbfe4413e4847ea0" -"checksum siphasher 0.2.3 (registry+https://github.com/rust-lang/crates.io-index)" = "0b8de496cf83d4ed58b6be86c3a275b8602f6ffe98d3024a869e124147a9a3ac" -"checksum slab 0.4.2 (registry+https://github.com/rust-lang/crates.io-index)" = "c111b5bd5695e56cffe5129854aa230b39c93a305372fdbb2668ca2394eea9f8" -"checksum smallvec 0.6.9 (registry+https://github.com/rust-lang/crates.io-index)" = "c4488ae950c49d403731982257768f48fada354a5203fe81f9bb6f43ca9002be" -"checksum socket2 0.3.9 (registry+https://github.com/rust-lang/crates.io-index)" = "4e626972d3593207547f14bf5fc9efa4d0e7283deb73fef1dff313dae9ab8878" -"checksum spin 0.5.0 (registry+https://github.com/rust-lang/crates.io-index)" = "44363f6f51401c34e7be73db0db371c04705d35efbe9f7d6082e03a921a32c55" -"checksum stable_deref_trait 1.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "dba1a27d3efae4351c8051072d619e3ade2820635c3958d826bfea39d59b54c8" -"checksum string 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)" = "b639411d0b9c738748b5397d5ceba08e648f4f1992231aa859af1a017f31f60b" -"checksum stringprep 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)" = "8ee348cb74b87454fff4b551cbf727025810a004f88aeacae7f85b87f4e9a1c1" -"checksum strsim 0.8.0 (registry+https://github.com/rust-lang/crates.io-index)" = "8ea5119cdb4c55b55d432abb513a0429384878c15dde60cc77b1c99de1a95a6a" -"checksum syn 0.15.34 (registry+https://github.com/rust-lang/crates.io-index)" = "a1393e4a97a19c01e900df2aec855a29f71cf02c402e2f443b8d2747c25c5dbe" -"checksum synstructure 0.10.2 (registry+https://github.com/rust-lang/crates.io-index)" = "02353edf96d6e4dc81aea2d8490a7e9db177bf8acb0e951c24940bf866cb313f" -"checksum tempfile 3.0.8 (registry+https://github.com/rust-lang/crates.io-index)" = "7dc4738f2e68ed2855de5ac9cdbe05c9216773ecde4739b2f095002ab03a13ef" -"checksum termcolor 1.0.4 (registry+https://github.com/rust-lang/crates.io-index)" = "4096add70612622289f2fdcdbd5086dc81c1e2675e6ae58d6c4f62a16c6d7f2f" -"checksum termion 1.5.2 (registry+https://github.com/rust-lang/crates.io-index)" = "dde0593aeb8d47accea5392b39350015b5eccb12c0d98044d856983d89548dea" -"checksum textwrap 0.11.0 (registry+https://github.com/rust-lang/crates.io-index)" = "d326610f408c7a4eb6f51c37c330e496b08506c9457c9d34287ecc38809fb060" -"checksum thread_local 0.3.6 (registry+https://github.com/rust-lang/crates.io-index)" = "c6b53e329000edc2b34dbe8545fd20e55a333362d0a321909685a19bd28c3f1b" -"checksum time 0.1.42 (registry+https://github.com/rust-lang/crates.io-index)" = "db8dcfca086c1143c9270ac42a2bbd8a7ee477b78ac8e45b19abfb0cbede4b6f" -"checksum tokio 0.1.20 (registry+https://github.com/rust-lang/crates.io-index)" = "94a1f9396aec29d31bb16c24d155cfa144d1af91c40740125db3131bdaf76da8" -"checksum tokio-buf 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "8fb220f46c53859a4b7ec083e41dec9778ff0b1851c0942b211edb89e0ccdc46" -"checksum tokio-current-thread 0.1.6 (registry+https://github.com/rust-lang/crates.io-index)" = "d16217cad7f1b840c5a97dfb3c43b0c871fef423a6e8d2118c604e843662a443" -"checksum tokio-executor 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)" = "83ea44c6c0773cc034771693711c35c677b4b5a4b21b9e7071704c54de7d555e" -"checksum tokio-io 0.1.12 (registry+https://github.com/rust-lang/crates.io-index)" = "5090db468dad16e1a7a54c8c67280c5e4b544f3d3e018f0b913b400261f85926" -"checksum tokio-reactor 0.1.9 (registry+https://github.com/rust-lang/crates.io-index)" = "6af16bfac7e112bea8b0442542161bfc41cbfa4466b580bdda7d18cb88b911ce" -"checksum tokio-sync 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)" = "5b2f843ffdf8d6e1f90bddd48da43f99ab071660cd92b7ec560ef3cdfd7a409a" -"checksum tokio-tcp 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)" = "1d14b10654be682ac43efee27401d792507e30fd8d26389e1da3b185de2e4119" -"checksum tokio-threadpool 0.1.14 (registry+https://github.com/rust-lang/crates.io-index)" = "72558af20be886ea124595ea0f806dd5703b8958e4705429dd58b3d8231f72f2" -"checksum tokio-timer 0.2.11 (registry+https://github.com/rust-lang/crates.io-index)" = "f2106812d500ed25a4f38235b9cae8f78a09edf43203e16e59c3b769a342a60e" -"checksum tokio-trace-core 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "350c9edade9830dc185ae48ba45667a445ab59f6167ef6d0254ec9d2430d9dd3" -"checksum try-lock 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)" = "e604eb7b43c06650e854be16a2a03155743d3752dd1c943f6829e26b7a36e382" -"checksum try_from 0.3.2 (registry+https://github.com/rust-lang/crates.io-index)" = "283d3b89e1368717881a9d51dad843cc435380d8109c9e47d38780a324698d8b" -"checksum typenum 1.10.0 (registry+https://github.com/rust-lang/crates.io-index)" = "612d636f949607bdf9b123b4a6f6d966dedf3ff669f7f045890d3a4a73948169" -"checksum ucd-util 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)" = "535c204ee4d8434478593480b8f86ab45ec9aae0e83c568ca81abf0fd0e88f86" -"checksum unicase 1.4.2 (registry+https://github.com/rust-lang/crates.io-index)" = "7f4765f83163b74f957c797ad9253caf97f103fb064d3999aea9568d09fc8a33" -"checksum unicase 2.4.0 (registry+https://github.com/rust-lang/crates.io-index)" = "a84e5511b2a947f3ae965dcb29b13b7b1691b6e7332cf5dbc1744138d5acb7f6" -"checksum unicode-bidi 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)" = "49f2bd0c6468a8230e1db229cff8029217cf623c767ea5d60bfbd42729ea54d5" -"checksum unicode-normalization 0.1.8 (registry+https://github.com/rust-lang/crates.io-index)" = "141339a08b982d942be2ca06ff8b076563cbe223d1befd5450716790d44e2426" -"checksum unicode-width 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)" = "882386231c45df4700b275c7ff55b6f3698780a650026380e72dabe76fa46526" -"checksum unicode-xid 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "fc72304796d0818e357ead4e000d19c9c174ab23dc11093ac919054d20a6a7fc" -"checksum untrusted 0.6.2 (registry+https://github.com/rust-lang/crates.io-index)" = "55cd1f4b4e96b46aeb8d4855db4a7a9bd96eeeb5c6a1ab54593328761642ce2f" -"checksum url 1.7.2 (registry+https://github.com/rust-lang/crates.io-index)" = "dd4e7c0d531266369519a4aa4f399d748bd37043b00bde1e4ff1f60a120b355a" -"checksum utf8-ranges 1.0.2 (registry+https://github.com/rust-lang/crates.io-index)" = "796f7e48bef87609f7ade7e06495a87d5cd06c7866e6a5cbfceffc558a243737" -"checksum uuid 0.7.4 (registry+https://github.com/rust-lang/crates.io-index)" = "90dbc611eb48397705a6b0f6e917da23ae517e4d127123d2cf7674206627d32a" -"checksum vcpkg 0.2.6 (registry+https://github.com/rust-lang/crates.io-index)" = "def296d3eb3b12371b2c7d0e83bfe1403e4db2d7a0bba324a12b21c4ee13143d" -"checksum vec_map 0.8.1 (registry+https://github.com/rust-lang/crates.io-index)" = "05c78687fb1a80548ae3250346c3db86a80a7cdd77bda190189f2d0a0987c81a" -"checksum version_check 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)" = "914b1a6776c4c929a602fafd8bc742e06365d4bcbe48c30f9cca5824f70dc9dd" -"checksum want 0.0.6 (registry+https://github.com/rust-lang/crates.io-index)" = "797464475f30ddb8830cc529aaaae648d581f99e2036a928877dfde027ddf6b3" -"checksum winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)" = "167dc9d6949a9b857f3451275e911c3f44255842c1f7a76f33c55103a909087a" -"checksum winapi 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)" = "f10e386af2b13e47c89e7236a7a14a086791a2b88ebad6df9bf42040195cf770" -"checksum winapi-build 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "2d315eee3b34aca4797b2da6b13ed88266e6d612562a0c46390af8299fc699bc" -"checksum winapi-i686-pc-windows-gnu 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)" = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6" -"checksum winapi-util 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)" = "7168bab6e1daee33b4557efd0e95d5ca70a03706d39fa5f3fe7a236f584b03c9" -"checksum winapi-x86_64-pc-windows-gnu 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)" = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" -"checksum wincolor 1.0.1 (registry+https://github.com/rust-lang/crates.io-index)" = "561ed901ae465d6185fa7864d63fbd5720d0ef718366c9a4dc83cf6170d7e9ba" -"checksum ws2_32-sys 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "d59cefebd0c892fa2dd6de581e937301d8552cb44489cdff035c6187cb63fa5e" +checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" diff --git a/indexer/Cargo.toml b/indexer/Cargo.toml index 6d952cb..271da67 100644 --- a/indexer/Cargo.toml +++ b/indexer/Cargo.toml @@ -2,19 +2,20 @@ name = "indexer" version = "0.1.0" authors = ["Yorhel <git@yorhel.nl>"] +edition = "2018" [dependencies] regex = "1.1.0" log = "0.4.1" -env_logger = "0.6.0" +env_logger = "0.9.0" lazy_static = "1.0.0" libc = "0.2.39" libarchive3-sys = "0.1.2" encoding = { git = "https://github.com/lifthrasiir/rust-encoding", features = ["no-optimized-legacy-encoding"] } -ring = "0.14.6" -postgres = "0.15.2" +ring = "0.16.20" +postgres = "0.19.0" clap = "2.31.2" -reqwest = "0.9.17" -url = "1.7.0" +ureq = "2.3.1" +percent-encoding = "2.0" chrono = "0.4.0" -quick-xml = "0.14.0" +quick-xml = "0.22.0" diff --git a/indexer/src/archive.rs b/indexer/src/archive.rs index 57c8305..75aba3b 100644 --- a/indexer/src/archive.rs +++ b/indexer/src/archive.rs @@ -1,6 +1,5 @@ use std::str; use std::ptr; -use std::error::Error as ErrorTrait; use std::io::{Result,Error,Read}; use std::ffi::{CStr,CString}; @@ -23,7 +22,7 @@ use libarchive3_sys::ffi; pub struct Archive<'a> { a: *mut ffi::Struct_archive, - rd: &'a mut Read, + rd: &'a mut dyn Read, buf: Vec<u8>, err: Option<Error>, eof: bool, @@ -61,7 +60,7 @@ unsafe extern "C" fn archive_read_cb(_: *mut ffi::Struct_archive, data: *mut c_v match arch.rd.read(&mut arch.buf[..]) { Ok(s) => s as ssize_t, Err(e) => { - let desc = CString::new(e.description()).unwrap(); + let desc = CString::new(e.to_string()).unwrap(); let fmt = CString::new("%s").unwrap(); ffi::archive_set_error(arch.a, e.raw_os_error().unwrap_or(0), fmt.as_ptr(), desc.as_ptr()); arch.err = Some(e); @@ -72,7 +71,7 @@ unsafe extern "C" fn archive_read_cb(_: *mut ffi::Struct_archive, data: *mut c_v impl<'a> Archive<'a> { - fn new(rd: &mut Read, a: *mut ffi::Struct_archive) -> Result<Box<Archive>> { + fn new(rd: &mut dyn Read, a: *mut ffi::Struct_archive) -> Result<Box<Archive>> { let bufsize = 64*1024; let mut buf = Vec::with_capacity(bufsize); unsafe { buf.set_len(bufsize) }; @@ -131,17 +130,17 @@ impl<'a> Archive<'a> { } } - pub fn open_archive(rd: &mut Read) -> Result<Option<ArchiveEntry>> { + pub fn open_archive(rd: &mut dyn Read) -> Result<Option<ArchiveEntry>> { let a = unsafe { let a = ffi::archive_read_new(); ffi::archive_read_support_filter_all(a); ffi::archive_read_support_format_all(a); a }; - try!(Self::new(rd, a)).entry() + Self::new(rd, a)?.entry() } - pub fn open_raw(rd: &mut Read) -> Result<RawEntry> { + pub fn open_raw(rd: &mut dyn Read) -> Result<RawEntry> { let a = unsafe { let a = ffi::archive_read_new(); ffi::archive_read_support_filter_all(a); @@ -149,7 +148,7 @@ impl<'a> Archive<'a> { ffi::archive_read_support_format_empty(a); a }; - let mut a = try!(Self::new(rd, a)); + let mut a = Self::new(rd, a)?; let mut e: *mut ffi::Struct_archive_entry = ptr::null_mut(); let res = unsafe { ffi::archive_read_next_header(a.a, &mut e) }; match res { @@ -282,10 +281,10 @@ pub fn walk<F>(ent: Option<ArchiveEntry>, mut cb: F) -> Result<()> { let mut ent = ent; while let Some(mut e) = ent { - if !try!(cb(&mut e)) { + if !cb(&mut e)? { break; } - ent = try!(e.next()); + ent = e.next()?; } Ok(()) } diff --git a/indexer/src/archread.rs b/indexer/src/archread.rs index d9c71f3..146ce79 100644 --- a/indexer/src/archread.rs +++ b/indexer/src/archread.rs @@ -1,7 +1,7 @@ use std::io::Result; use std::collections::HashMap; -use archive::{walk,ArchiveEntry,FileType}; +use crate::archive::{walk,ArchiveEntry,FileType}; /* I had hoped that reading man pages from an archive would just be a simple: * @@ -121,7 +121,7 @@ impl FileList { links: Vec::new(), }; - try!(walk(ent, |mut e| { + walk(ent, |mut e| { let path = match e.path() { Some(x) => x.to_string(), None => { warn!("Invalid UTF-8 filename in archive"); return Ok(true) } @@ -144,7 +144,7 @@ impl FileList { EntryType::Hardlink } else if interest_cb(&path) { let pathv = [&path as &str]; - try!(file_cb(&pathv[..], &mut e)); + file_cb(&pathv[..], &mut e)?; EntryType::Handled } else { EntryType::Regular @@ -157,7 +157,7 @@ impl FileList { let opath = ocomp.join("/"); if interest_cb(&opath) { let pathv = [&opath as &str]; - try!(file_cb(&pathv[..], &mut e)); + file_cb(&pathv[..], &mut e)?; *fl.seen.get_mut(&opath).unwrap() = EntryType::Handled; } } @@ -173,7 +173,7 @@ impl FileList { fl.seen.insert(path, et); Ok(true) - })); + })?; Ok(fl) } @@ -301,7 +301,7 @@ impl MissedFiles { walk(ent, |mut e| { if let Some(f) = e.path().and_then(|p| self.0.remove(p)) { let v: Vec<&str> = f.iter().map(|x| x as &str).collect(); - try!(file_cb(&v, &mut e)) + file_cb(&v, &mut e)? } Ok(self.0.len() > 0) }) diff --git a/indexer/src/main.rs b/indexer/src/main.rs index 2373723..f8c1933 100644 --- a/indexer/src/main.rs +++ b/indexer/src/main.rs @@ -1,23 +1,13 @@ #[macro_use] extern crate log; #[macro_use] extern crate lazy_static; #[macro_use] extern crate clap; -extern crate env_logger; -extern crate regex; -extern crate libarchive3_sys; -extern crate libc; -extern crate ring; -extern crate encoding; -extern crate postgres; -extern crate reqwest; -extern crate url; -extern crate chrono; -extern crate quick_xml; mod archive; mod archread; mod man; mod open; mod pkg; +mod sys_alpine; mod sys_arch; mod sys_deb; mod sys_freebsd1; @@ -27,12 +17,12 @@ mod sys_rpm; // Convenience function to get a system id by short-name. Panics if the system doesn't exist. -fn sysbyshort(conn: &postgres::GenericConnection, short: &str) -> i32 { - let r = conn.query("SELECT id FROM systems WHERE short = $1", &[&short]).unwrap(); - if r.is_empty() { +fn sysbyshort<T>(conn: &mut T, short: &str) -> i32 where T: postgres::GenericClient { + if let Some(r) = conn.query_opt("SELECT id FROM systems WHERE short = $1", &[&short]).unwrap() { + return r.get(0); + } else { panic!("Invalid system: {}", short); } - r.get(0).get(0) } @@ -52,6 +42,12 @@ fn main() { (@arg arch: --arch +takes_value "Architecture") (@arg FILE: +required "Package file") ) + (@subcommand alpine => + (about: "Index an Alpine Linux repository") + (@arg sys: --sys +required +takes_value "System short-name") + (@arg mirror: --mirror +required +takes_value "Mirror URL") + (@arg repo: --repo +required +takes_value "Repository name") + ) (@subcommand arch => (about: "Index an Arch Linux repository") (@arg sys: --sys +required +takes_value "System short-name") @@ -112,7 +108,7 @@ fn main() { Ok(x) => x, Err(_) => { error!("MANNED_PG not set."); return } }; - let db = match postgres::Connection::connect(&dbhost[..], postgres::TlsMode::None) { + let mut db = match postgres::Client::connect(&dbhost[..], postgres::tls::NoTls) { Ok(x) => x, Err(x) => { error!("Can't connect to postgres: {}", x); return }, }; @@ -125,9 +121,10 @@ fn main() { "max" => pkg::Date::Max, s => pkg::Date::Known(s), }; - pkg::pkg(&db, pkg::PkgOpt { + let sys = sysbyshort(&mut db, matches.value_of("sys").unwrap()); + pkg::pkg(&mut db, pkg::PkgOpt { force: matches.is_present("force"), - sys: sysbyshort(&db, matches.value_of("sys").unwrap()), + sys: sys, cat: matches.value_of("cat").unwrap(), pkg: matches.value_of("pkg").unwrap(), ver: matches.value_of("ver").unwrap(), @@ -137,17 +134,25 @@ fn main() { }); } + if let Some(matches) = arg.subcommand_matches("alpine") { + let sys = sysbyshort(&mut db, matches.value_of("sys").unwrap()); + sys_alpine::sync(&mut db, sys, + matches.value_of("mirror").unwrap(), + matches.value_of("repo").unwrap() + ); + } + if let Some(matches) = arg.subcommand_matches("arch") { - sys_arch::sync(&db, - sysbyshort(&db, matches.value_of("sys").unwrap()), + let sys = sysbyshort(&mut db, matches.value_of("sys").unwrap()); + sys_arch::sync(&mut db, sys, matches.value_of("mirror").unwrap(), matches.value_of("repo").unwrap() ); } if let Some(matches) = arg.subcommand_matches("deb") { - sys_deb::sync(&db, - sysbyshort(&db, matches.value_of("sys").unwrap()), + let sys = sysbyshort(&mut db, matches.value_of("sys").unwrap()); + sys_deb::sync(&mut db, sys, matches.value_of("mirror").unwrap(), matches.value_of("contents").map(|e| { open::Path{ path: e, cache: true, canbelocal: true} }), open::Path{ path: matches.value_of("packages").unwrap(), cache: true, canbelocal: true}, @@ -155,31 +160,31 @@ fn main() { } if let Some(matches) = arg.subcommand_matches("freebsd1") { - sys_freebsd1::sync(&db, - sysbyshort(&db, matches.value_of("sys").unwrap()), + let sys = sysbyshort(&mut db, matches.value_of("sys").unwrap()); + sys_freebsd1::sync(&mut db, sys, matches.value_of("arch").unwrap(), matches.value_of("mirror").unwrap() ).unwrap_or_else(|e| error!("{}", e)); } if let Some(matches) = arg.subcommand_matches("freebsd2") { - sys_freebsd2::sync(&db, - sysbyshort(&db, matches.value_of("sys").unwrap()), + let sys = sysbyshort(&mut db, matches.value_of("sys").unwrap()); + sys_freebsd2::sync(&mut db, sys, matches.value_of("mirror").unwrap() ).unwrap_or_else(|e| error!("{}", e)); } if let Some(matches) = arg.subcommand_matches("rpmdir") { - sys_rpmdir::sync(&db, - sysbyshort(&db, matches.value_of("sys").unwrap()), + let sys = sysbyshort(&mut db, matches.value_of("sys").unwrap()); + sys_rpmdir::sync(&mut db, sys, matches.value_of("cat").unwrap(), matches.value_of("mirror").unwrap() ).unwrap_or_else(|e| error!("{}", e)); } if let Some(matches) = arg.subcommand_matches("rpm") { - sys_rpm::sync(&db, - sysbyshort(&db, matches.value_of("sys").unwrap()), + let sys = sysbyshort(&mut db, matches.value_of("sys").unwrap()); + sys_rpm::sync(&mut db, sys, matches.value_of("cat").unwrap(), matches.value_of("mirror").unwrap() ).unwrap_or_else(|e| error!("{}", e)); diff --git a/indexer/src/man.rs b/indexer/src/man.rs index 32805ae..d594d81 100644 --- a/indexer/src/man.rs +++ b/indexer/src/man.rs @@ -8,7 +8,7 @@ use encoding::{all,EncodingRef}; use encoding::label::encoding_from_whatwg_label; use ring::digest; -use archive::Archive; +use crate::archive::Archive; // Anything larger than this just isn't a man page. I hope. const MAX_MAN_SIZE: u64 = 20*1024*1024; @@ -27,29 +27,39 @@ pub fn parse_path(path: &str) -> Option<(&str, &str, &str)> { /man[a-z0-9]/ # Subdir ([^/]+?) # Man page name (non-greedy) \. ([^/\.]+) # Section - (?: \. (?: gz|lzma|bz2|xz ))* $ # Any number of compression extensions + (?: \. (?: gz|lzma|bz2|xz|zst ))* $ # Any number of compression extensions ").unwrap(); } let cap = match RE.captures(path) { Some(x) => x, None => return None }; - let locale = cap.get(1).map(|e| e.as_str()).unwrap_or(""); + let mut locale = cap.get(1).map(|e| e.as_str()).unwrap_or(""); let name = cap.get(2).unwrap().as_str(); let section = cap.get(3).unwrap().as_str(); + // Some weird directories that happen to match the locale + if locale.contains("openmpi") || locale.contains("mpich") || locale.contains("mvapich") { + locale = ""; + } + // Not everything matching the regex is necessarily a man page, exclude some special cases. match (name, section, locale) { // Files that totally aren't man pages ("Makefile", "am", _) | (".cvsignore", _, _) | (_, "in", _) | - (_, "gz", _) | + (_, "orig", _) | + (_, "pdf", _) | + (_, "md", _) | (_, "lzma", _) | + (_, "gz", _) | + (_, "gzip", _) | (_, "bz2", _) | (_, "xz", _) | (_, "html", _) => None, - // Some weird directories that happen to match the locale + // Some more weird directories that happen to match the locale (n, s, "5man") | (n, s, "c") | + (n, s, "man") | (n, s, "man1") | (n, s, "man2") | (n, s, "man3") | @@ -59,7 +69,8 @@ pub fn parse_path(path: &str) -> Option<(&str, &str, &str)> { (n, s, "man7") | (n, s, "man8") | (n, s, "Man-Part1") | - (n, s, "Man-Part2") => Some((n, s, "")), + (n, s, "Man-Part2") | + (n, s, "overrides") => Some((n, s, "")), // Nothing special! x => Some(x) } @@ -195,16 +206,16 @@ fn codec_from_path(path: &str) -> Option<EncodingRef> { // Decompresses / decodes a man page and returns its SHA-1 hash, encoding name, and UTF-8 contents. -pub fn decode(paths: &[&str], ent: &mut Read) -> io::Result<(digest::Digest,&'static str,String)> { - let mut decomp = try!(Archive::open_raw(ent)).take(MAX_MAN_SIZE+1); +pub fn decode(paths: &[&str], ent: &mut dyn Read) -> io::Result<(digest::Digest,&'static str,String)> { + let mut decomp = Archive::open_raw(ent)?.take(MAX_MAN_SIZE+1); let mut data = Vec::new(); - try!(decomp.read_to_end(&mut data)); + decomp.read_to_end(&mut data)?; if let Some(e) = validate(&data) { return Err(io::Error::new(io::ErrorKind::InvalidData, e)); } - let dig = digest::digest(&digest::SHA1, &data); + let dig = digest::digest(&digest::SHA1_FOR_LEGACY_USE_ONLY, &data); // Create a list of encodings to try, starting with UTF-8 let mut encs : Vec<EncodingRef> = vec![all::UTF_8]; @@ -301,6 +312,6 @@ fn test_decode_zh() { assert_eq!(dig.as_ref(), &filehash[..]); assert_eq!(enc, "gbk"); - let utf8dig = digest::digest(&digest::SHA1, s.as_bytes()); + let utf8dig = digest::digest(&digest::SHA1_FOR_LEGACY_USE_ONLY, s.as_bytes()); assert_eq!(utf8dig.as_ref(), &utf8hash[..]); } diff --git a/indexer/src/open.rs b/indexer/src/open.rs index 9ecdfa4..9375d5a 100644 --- a/indexer/src/open.rs +++ b/indexer/src/open.rs @@ -3,9 +3,8 @@ use std::fs::{File,create_dir_all,metadata,read_dir,remove_file}; use std::time::{Duration,SystemTime}; use regex::bytes::Regex; use ring::digest; -use url::Url; -use url::percent_encoding::percent_decode; -use reqwest; +use percent_encoding::percent_decode; +use ureq; const CACHE_PATH: &'static str = "/var/tmp/manned-indexer"; @@ -20,34 +19,20 @@ pub struct Path<'a> { } -fn cache_fn(url: &Url) -> String { - let name = url.path_segments().unwrap().last().unwrap(); - let name = if name == "" { "index" } else { name }; - - let hash = digest::digest(&digest::SHA1, url.as_str().as_bytes()) - .as_ref()[0..8].into_iter() - .fold(0u64, |a, &e| (a<<8) + e as u64); - - format!("{}/{}-{}-{:x}", CACHE_PATH, url.host_str().unwrap(), name, hash) -} - - -fn fetch(url: &str) -> Result<Box<Read>> { - let res = try!(reqwest::Client::new() - .get(url) - .header("User-Agent", "Man page crawler (info@manned.org; https://manned.org/)") - .send() - .map_err(|e| Error::new(ErrorKind::Other, format!("Reqwest: {}", e))) - ); - if !res.status().is_success() { +fn fetch(url: &str) -> Result<Box<dyn Read>> { + let res = ureq::get(url) + .set("User-Agent", "Man page crawler (info@manned.org; https://manned.org/)") + .call() + .map_err(|e| Error::new(ErrorKind::Other, format!("Ureq: {}", e)))?; + if res.status() != 200 { return Err(Error::new(ErrorKind::Other, format!("HTTP: {}", res.status()) )); } - Ok(Box::new(res) as Box<Read>) + Ok(Box::new(res.into_reader()) as Box<dyn Read>) } -fn file(path: &str) -> Result<Box<Read>> { - Ok(Box::new(try!(File::open(path))) as Box<Read>) +fn file(path: &str) -> Result<Box<dyn Read>> { + Ok(Box::new(File::open(path)?) as Box<dyn Read>) } @@ -65,28 +50,27 @@ pub fn clear_cache() -> Result<()> { impl<'a> Path<'a> { - pub fn open(&self) -> Result<Box<Read>> { - if let Ok(url) = Url::parse(self.path) { - if url.scheme() != "http" { - return Err(Error::new(ErrorKind::Other, "Invalid scheme")); - } - + pub fn open(&self) -> Result<Box<dyn Read>> { + if self.path.starts_with("http://") || self.path.starts_with("https://") { if self.cache { - let cfn = cache_fn(&url); + let hash = digest::digest(&digest::SHA256, self.path.as_bytes()) + .as_ref()[0..8].into_iter() + .fold(0u64, |a, &e| (a<<8) + e as u64); + + let cfn = format!("{}/{:x}", CACHE_PATH, hash); if let Ok(f) = file(&cfn) { return Ok(f); } { - let mut rd = try!(fetch(url.as_str())); - let mut wr = try!(File::create(&cfn)); - try!(copy(&mut rd, &mut wr)); + let mut rd = fetch(self.path)?; + let mut wr = File::create(&cfn)?; + copy(&mut rd, &mut wr)?; } file(&cfn) } else { - fetch(url.as_str()) + fetch(self.path) } - } else if self.canbelocal { file(self.path) diff --git a/indexer/src/pkg.rs b/indexer/src/pkg.rs index 93b7c14..11aab66 100644 --- a/indexer/src/pkg.rs +++ b/indexer/src/pkg.rs @@ -3,10 +3,10 @@ use std::io::{Error,ErrorKind,Read}; use postgres; use chrono::NaiveDateTime; -use open; -use archread; -use man; -use archive::{Format,Archive,ArchiveEntry}; +use crate::open; +use crate::archread; +use crate::man; +use crate::archive::{Format,Archive,ArchiveEntry}; pub static mut DRY_RUN: bool = false; @@ -48,23 +48,27 @@ pub struct PkgOpt<'a> { } -fn insert_pkg(tr: &postgres::transaction::Transaction, opt: &PkgOpt) -> Option<i32> { +fn insert_pkg(tr: &mut postgres::Transaction, opt: &PkgOpt) -> Option<i32> { let pkginfo = format!("sys {} / {} / {} - {} @ {:?} @ {}", opt.sys, opt.cat, opt.pkg, opt.ver, opt.date, opt.file.path); - // The ON CONFLICT .. DO UPDATE is used instead of DO NOTHING because in that case the - // RETURNING clause wouldn't give us a package id. - let q = "INSERT INTO packages (system, category, name) VALUES($1, $2, $3) - ON CONFLICT ON CONSTRAINT packages_system_name_category_key DO UPDATE SET name=$3 RETURNING id"; - let pkgid: i32 = match tr.query(q, &[&opt.sys, &opt.cat, &opt.pkg]) { + // Use a custom CTE-based insert-or-update. Using an INSERT with an ON CONFLICT clause would be + // easier, but has the downside of allocating a new package id even if one already exists. + // The separate UPDATE query makes sure to unflag the package as dead while not causing any + // database writes when the row's already fine. + let q = "WITH p(id) AS (SELECT id FROM packages WHERE system = $1 AND category = $2 AND name = $3), + u AS (UPDATE packages SET dead = FALSE FROM p WHERE packages.id = p.id AND dead), + i(id) AS (INSERT INTO packages (system, category, name) SELECT $1, $2, $3 WHERE NOT EXISTS(SELECT 1 FROM p) RETURNING id) + SELECT id FROM p UNION SELECT id FROM i"; + let pkgid: i32 = match tr.query_one(q, &[&opt.sys, &opt.cat, &opt.pkg]) { Err(e) => { error!("Can't insert package in database: {}", e); return None; }, - Ok(r) => r.get(0).get(0), + Ok(r) => r.get(0), }; let q = "SELECT id FROM package_versions WHERE package = $1 AND version = $2"; - let res = tr.query(q, &[&pkgid, &opt.ver]).unwrap(); + let res = tr.query_opt(q, &[&pkgid, &opt.ver]).unwrap(); let verid : i32; @@ -73,33 +77,41 @@ fn insert_pkg(tr: &postgres::transaction::Transaction, opt: &PkgOpt) -> Option<i _ => "1980-01-01", // Placeholder }; - if res.is_empty() { + if res.is_none() { let q = "INSERT INTO package_versions (package, version, released, arch) VALUES($1, $2, $3::text::date, $4) RETURNING id"; - verid = tr.query(q, &[&pkgid, &opt.ver, &date, &opt.arch]).unwrap().get(0).get(0); + verid = tr.query_one(q, &[&pkgid, &opt.ver, &date, &opt.arch]).unwrap().get(0); info!("New package pkgid {} verid {}, {}", pkgid, verid, pkginfo); Some(verid) } else if opt.force { // XXX: Should we update released & arch here? - verid = res.get(0).get(0); + verid = res?.get(0); info!("Overwriting package pkgid {} verid {}, {}", pkgid, verid, pkginfo); - tr.query("DELETE FROM man WHERE package = $1", &[&verid]).unwrap(); + tr.query("DELETE FROM files WHERE pkgver = $1", &[&verid]).unwrap(); Some(verid) } else { - debug!("Package already in database, pkgid {} verid {}, {}", pkgid, res.get(0).get::<usize,i32>(0), pkginfo); + debug!("Package already in database, pkgid {} verid {}, {}", pkgid, res?.get::<usize,i32>(0), pkginfo); None } } -fn insert_man_row(tr: &postgres::GenericConnection, verid: i32, path: &str, enc: &str, hash: &[u8]) { +fn insert_man_row<T: postgres::GenericClient>(tr: &mut T, verid: i32, path: &str, enc: &str, content: i32) { let (name, sect, locale) = man::parse_path(path).unwrap(); - let locale = if locale == "" { None } else { Some(locale) }; - if let Err(e) = tr.execute( - "INSERT INTO man (package, name, filename, locale, hash, section, encoding) VALUES ($1, $2, '/'||$3, $4, $5, $6, $7)", - &[&verid, &name, &path, &locale, &hash, §, &enc] - ) { + let q = "WITH ms(id) AS (SELECT id FROM mans WHERE name = $2 AND section = $3), + mi(id) AS (INSERT INTO mans (name, section) SELECT $2, $3 WHERE NOT EXISTS(SELECT 1 FROM ms) RETURNING id), + m(id) AS (SELECT id FROM ms UNION SELECT id FROM mi), + ls(id) AS (SELECT id FROM locales WHERE locale = $5), + li(id) AS (INSERT INTO locales (locale) SELECT $5 WHERE NOT EXISTS(SELECT 1 FROM ls) RETURNING id), + l(id) AS (SELECT id FROM ls UNION SELECT id FROM li), + es(id) AS (SELECT id FROM encodings WHERE encoding = $6), + ei(id) AS (INSERT INTO encodings (encoding) SELECT $6 WHERE NOT EXISTS(SELECT 1 FROM es) RETURNING id), + e(id) AS (SELECT id FROM es UNION SELECT id FROM ei), + c(shorthash) AS (SELECT hash_to_shorthash(hash) FROM contents WHERE id = $4) + INSERT INTO files (pkgver, man, content, shorthash, locale, encoding, filename) + SELECT $1, m.id, $4, c.shorthash, l.id, e.id, '/'||$7 FROM m, l, e, c"; + if let Err(e) = tr.execute(q, &[&verid, &name, §, &content, &locale, &enc, &path]) { // I think this can only happen if archread gives us the same file twice, which really // shouldn't happen. But I'd rather continue with an error logged than panic. error!("Can't insert verid {} fn {}: {}", verid, path, e); @@ -107,7 +119,7 @@ fn insert_man_row(tr: &postgres::GenericConnection, verid: i32, path: &str, enc: } -fn insert_man(tr: &postgres::GenericConnection, verid: i32, paths: &[&str], ent: &mut Read) { +fn insert_man<T: postgres::GenericClient>(tr: &mut T, verid: i32, paths: &[&str], ent: &mut dyn Read) { let (dig, enc, mut cont) = match man::decode(paths, ent) { Err(e) => { error!("Error decoding {}: {}", paths[0], e); return }, Ok(x) => x, @@ -119,27 +131,30 @@ fn insert_man(tr: &postgres::GenericConnection, verid: i32, paths: &[&str], ent: cont = cont.replace(0 as char, ""); } - tr.execute( - "INSERT INTO contents (hash, content) VALUES($1, $2) ON CONFLICT (hash) DO NOTHING", - &[&dig.as_ref(), &cont] - ).unwrap(); + let q = "WITH s(id) AS (SELECT id FROM contents WHERE hash = $1), + i(id) AS (INSERT INTO contents (hash, content) SELECT $1, $2 WHERE NOT EXISTS(SELECT 1 FROM s) RETURNING id) + SELECT id FROM s UNION SELECT id FROM i"; + let id: i32 = tr.query_one(q, &[&dig.as_ref(), &cont]).unwrap().get(0); for path in paths { - insert_man_row(tr, verid, path, enc, dig.as_ref()); + insert_man_row(tr, verid, path, enc, id); info!("Inserted man page: {} ({})", path, enc); } } -fn insert_link(tr: &postgres::GenericConnection, verid: i32, src: &str, dest: &str) { - let res = tr.query("SELECT hash, encoding FROM man WHERE package = $1 AND filename = '/'||$2", &[&verid, &dest]).unwrap(); - if res.is_empty() { /* Can happen if man::decode() failed previously. */ - error!("Link to unindexed man page: {} -> {}", src, dest); - return; - } - let hash: Vec<u8> = res.get(0).get(0); - let enc: String = res.get(0).get(1); - insert_man_row(tr, verid, src, &enc, &hash); +fn insert_link<T>(tr: &mut T, verid: i32, src: &str, dest: &str) where T: postgres::GenericClient { + let q = "SELECT f.content, e.encoding FROM files f JOIN encodings e ON e.id = f.encoding WHERE pkgver = $1 AND filename = '/'||$2"; + let res = match tr.query_opt(q, &[&verid, &dest]).unwrap() { + None => { /* Can happen if man::decode() failed previously. */ + error!("Link to unindexed man page: {} -> {}", src, dest); + return; + }, + Some(x) => x + }; + let content: i32 = res.get(0); + let enc: String = res.get(1); + insert_man_row(tr, verid, src, &enc, content); info!("Inserted man link: {} -> {}", src, dest); } @@ -173,19 +188,20 @@ fn with_pkg<F,T>(opt: &mut PkgOpt, cb: F) -> std::io::Result<T> } -fn index_pkg(tr: &postgres::GenericConnection, mut opt: PkgOpt, verid: i32) -> std::io::Result<()> { - let indexfunc = |paths: &[&str], ent: &mut ArchiveEntry| { - insert_man(tr, verid, paths, ent); - Ok(()) /* Don't propagate errors, continue handling other man pages */ - }; - +fn index_pkg<T: postgres::GenericClient>(tr: &mut T, mut opt: PkgOpt, verid: i32) -> std::io::Result<()> { let missed = with_pkg(&mut opt, |e, opt| { - archread::FileList::read(e, man::ismanpath, |ent| opt.date.update(ent), &indexfunc) + archread::FileList::read(e, man::ismanpath, |ent| opt.date.update(ent), |paths, ent| { + insert_man(tr, verid, paths, ent); + Ok(()) + }) })?.links(|src, dest| { insert_link(tr, verid, src, dest) }); if let Some(missed) = missed { warn!("Some links were missed, reading package again"); - with_pkg(&mut opt, |e, _| { missed.read(e, indexfunc) })? + with_pkg(&mut opt, |e, _| { missed.read(e, |paths, ent| { + insert_man(tr, verid, paths, ent); + Ok(()) + }) })? } match opt.date { @@ -201,21 +217,20 @@ fn index_pkg(tr: &postgres::GenericConnection, mut opt: PkgOpt, verid: i32) -> s } -pub fn pkg(conn: &postgres::GenericConnection, opt: PkgOpt) { - let tr = conn.transaction().unwrap(); - tr.set_rollback(); +pub fn pkg<T>(conn: &mut T, opt: PkgOpt) where T: postgres::GenericClient { + let mut tr = conn.transaction().unwrap(); - let verid = match insert_pkg(&tr, &opt) { Some(x) => x, None => return }; + let verid = match insert_pkg(&mut tr, &opt) { Some(x) => x, None => return }; if unsafe { DRY_RUN } { return; } - match index_pkg(&tr, opt, verid) { - Err(e) => error!("Error reading package: {}", e), - Ok(_) => tr.set_commit() + if let Err(e) = index_pkg(&mut tr, opt, verid) { + error!("Error reading package: {}", e); + return; } - if let Err(e) = tr.finish() { + if let Err(e) = tr.commit() { error!("Error finishing transaction: {}", e); } } diff --git a/indexer/src/sys_alpine.rs b/indexer/src/sys_alpine.rs new file mode 100644 index 0000000..241ca3e --- /dev/null +++ b/indexer/src/sys_alpine.rs @@ -0,0 +1,99 @@ +use std::str::FromStr; +use std::io::{Read,BufRead,BufReader}; +use postgres; + +use crate::archive; +use crate::open; +use crate::pkg; + +// https://git.alpinelinux.org/apk-tools/tree/doc/apk-repositories.5.scd +// https://git.alpinelinux.org/apk-tools/tree/src/package.c#n874 (apk_pkg_write_index_entry) + +pub fn read_index<T: postgres::GenericClient, R: Read>(pg: &mut T, sys: i32, mirror: &str, repo: &str, lst: R) { + let rd = BufReader::new(lst); + + let mut name = None; + let mut version = None; + let mut builddate = None; + let mut arch = None; + let mut lineno: u32 = 0; + + for line in rd.lines() { + lineno += 1; + let line = match line { + Err(e) => { error!("Can't read package index: {}", e); return }, + Ok(x) => x, + }; + + if line.starts_with("P:") { + name = Some(line[2..].to_string()); + } else if line.starts_with("V:") { + version = Some(line[2..].to_string()); + } else if line.starts_with("t:") { + builddate = i64::from_str(&line[2..]).ok(); + } else if line.starts_with("A:") { + arch = Some(line[2..].to_string()); + } + if line != "" { + continue; + } + + if name.is_none() || version.is_none() { + warn!("Package without name or version on line {}", lineno); + return; + } + + let pname = name.as_ref().unwrap(); + let pver = version.as_ref().unwrap(); + if pname == "man-pages" || pname.ends_with("-doc") { + let p = format!("{}/{}/x86_64/{}-{}.apk", mirror, repo, pname, pver); + pkg::pkg(pg, pkg::PkgOpt{ + force: false, + sys: sys, + cat: repo, + pkg: pname, + ver: pver, + date: builddate.map(pkg::Date::Found).unwrap_or(pkg::Date::Max), + arch: arch.as_deref(), + file: open::Path{ + path: &p, + cache: false, + canbelocal: false, + }, + }); + } + + name = None; + version = None; + builddate = None; + arch = None; + } +} + + +pub fn sync<T: postgres::GenericClient>(pg: &mut T, sys: i32, mirror: &str, repo: &str) { + info!("Reading packages from {} {}", mirror, repo); + + let path = format!("{}/{}/x86_64/APKINDEX.tar.gz", mirror, repo); + let path = open::Path{ path: &path, cache: true, canbelocal: false }; + let mut index = match path.open() { + Err(e) => { error!("Can't read package index: {}", e); return }, + Ok(x) => x, + }; + + let ent = match archive::Archive::open_archive(&mut index) { + Err(e) => { error!("Can't read package index: {}", e); return }, + Ok(x) => x, + }; + + let r = archive::walk(ent, |x| { + if x.path() == Some("APKINDEX") { + read_index(pg, sys, mirror, repo, x); + } + Ok(true) + }); + + if let Err(e) = r { + error!("Error reading package index: {}", e); + } +} diff --git a/indexer/src/sys_arch.rs b/indexer/src/sys_arch.rs index ee17266..4c0ea41 100644 --- a/indexer/src/sys_arch.rs +++ b/indexer/src/sys_arch.rs @@ -1,13 +1,14 @@ use std::str::FromStr; use std::io::{Read,BufRead,BufReader,Result}; +use std::collections::HashSet; use regex::Regex; use chrono::NaiveDateTime; use postgres; -use archive; -use open; -use man; -use pkg; +use crate::archive; +use crate::open; +use crate::man; +use crate::pkg; struct Meta { @@ -22,8 +23,7 @@ struct Meta { fn read_files<T: Read>(lst: T) -> Result<bool> { let rd = BufReader::new(lst); for line in rd.lines() { - let line = try!(line); - if man::ismanpath(&line) { + if man::ismanpath(&line?) { return Ok(true); } } @@ -33,7 +33,7 @@ fn read_files<T: Read>(lst: T) -> Result<bool> { fn read_desc(rd: &mut archive::ArchiveEntry) -> Result<Option<Meta>> { let mut data = String::new(); - try!(rd.take(64*1024).read_to_string(&mut data)); + rd.take(64*1024).read_to_string(&mut data)?; let path = rd.path().unwrap(); lazy_static! { @@ -75,7 +75,7 @@ fn read_desc(rd: &mut archive::ArchiveEntry) -> Result<Option<Meta>> { } -pub fn sync(pg: &postgres::GenericConnection, sys: i32, mirror: &str, repo: &str) { +pub fn sync<T: postgres::GenericClient>(pg: &mut T, sys: i32, mirror: &str, repo: &str) { info!("Reading packages from {} {}", mirror, repo); let path = format!("{}/{}/os/x86_64/{1:}.files.tar.gz", mirror, repo); @@ -92,14 +92,15 @@ pub fn sync(pg: &postgres::GenericConnection, sys: i32, mirror: &str, repo: &str let mut hasman = false; let mut meta = None; + let mut allpkgs = HashSet::new(); let r = archive::walk(ent, |x| { if x.filetype() == archive::FileType::Directory { hasman = false; meta = None; } else if x.path().unwrap().ends_with("/files") { - hasman = try!(read_files(x)); + hasman = read_files(x)?; } else if x.path().unwrap().ends_with("/desc") { - meta = try!(read_desc(x)); + meta = read_desc(x)?; } if hasman && meta.is_some() { @@ -121,6 +122,7 @@ pub fn sync(pg: &postgres::GenericConnection, sys: i32, mirror: &str, repo: &str canbelocal: false, }, }); + allpkgs.insert(m.name.into_boxed_str()); } Ok(true) @@ -129,4 +131,30 @@ pub fn sync(pg: &postgres::GenericConnection, sys: i32, mirror: &str, repo: &str if let Err(e) = r { error!("Error reading package index: {}", e); } + mark_dead(pg, sys, repo, allpkgs); +} + +fn mark_dead<T: postgres::GenericClient>(pg: &mut T, sys: i32, repo: &str, pkgs: HashSet<Box<str>>) { + let mut dead = Vec::new(); + for row in pg.query("SELECT id, name FROM packages WHERE system = $1 AND category = $2 AND NOT dead", &[&sys,&repo]).unwrap() { + let id: i32 = row.get(0); + let name: &str = row.get(1); + if !pkgs.contains(name) { + info!("Package not available in database anymore, marking dead; sys {} / {} / pkg {} ({})", sys, repo, id, name); + dead.push(id); + } + } + if dead.is_empty() { + return; + } + + let mut tr = pg.transaction().unwrap(); + let q = tr.prepare("UPDATE packages SET dead = TRUE WHERE id = $1").unwrap(); + for id in dead { + tr.execute(&q, &[&id]).unwrap(); + } + + if let Err(e) = tr.commit() { + error!("Error finishing transaction: {}", e); + } } diff --git a/indexer/src/sys_deb.rs b/indexer/src/sys_deb.rs index 46291bc..e92a7b7 100644 --- a/indexer/src/sys_deb.rs +++ b/indexer/src/sys_deb.rs @@ -5,10 +5,10 @@ use postgres; use regex; use regex::bytes::Regex; -use man; -use pkg; -use open; -use archive; +use crate::man; +use crate::pkg; +use crate::open; +use crate::archive; // Reference: https://wiki.debian.org/RepositoryFormat @@ -56,7 +56,7 @@ struct Pkg { } -fn handlepkg(pg: &postgres::GenericConnection, sys: i32, mirror: &str, manpkgs: &HashSet<String>, pkg: &Pkg) { +fn handlepkg<T: postgres::GenericClient>(pg: &mut T, sys: i32, mirror: &str, manpkgs: &HashSet<String>, pkg: &Pkg) { let name = match pkg.name { Some(ref x) => x, None => return }; if manpkgs.len() > 0 && !manpkgs.contains(name) { return @@ -94,7 +94,7 @@ fn handlepkg(pg: &postgres::GenericConnection, sys: i32, mirror: &str, manpkgs: } -pub fn sync(pg: &postgres::GenericConnection, sys: i32, mirror: &str, contents: Option<open::Path>, packages: open::Path) { +pub fn sync<T: postgres::GenericClient >(pg: &mut T, sys: i32, mirror: &str, contents: Option<open::Path>, packages: open::Path) { let manpkgs = match get_contents(contents) { Err(e) => { error!("Can't read {}: {}", contents.unwrap().path, e); return }, Ok(x) => x, diff --git a/indexer/src/sys_freebsd1.rs b/indexer/src/sys_freebsd1.rs index f2e9e29..66c30b1 100644 --- a/indexer/src/sys_freebsd1.rs +++ b/indexer/src/sys_freebsd1.rs @@ -3,8 +3,8 @@ use std::io::Result; use regex::Regex; use postgres; -use open; -use pkg; +use crate::open; +use crate::pkg; // Sync a FreeBSD <= 9.2 package respository. @@ -26,7 +26,7 @@ use pkg; // 'pear-PHPUnit version 1.3.3', because there is a 'pear' package in 'Latest' but no // 'pear-PHPUnit'. This is handled with a static list of package names to add to the 'pkgs' list, // see EXTRA_PKGS below. -pub fn sync(pg: &postgres::GenericConnection, sys: i32, arch: &str, mirror: &str) -> Result<()> { +pub fn sync<T: postgres::GenericClient>(pg: &mut T, sys: i32, arch: &str, mirror: &str) -> Result<()> { let path = format!("{}Latest/", mirror); let mut pkgs : Vec<String> = open::Path{path: &path, cache: true, canbelocal: false} .dirlist()?.into_iter() diff --git a/indexer/src/sys_freebsd2.rs b/indexer/src/sys_freebsd2.rs index 193d7b0..9b42c8c 100644 --- a/indexer/src/sys_freebsd2.rs +++ b/indexer/src/sys_freebsd2.rs @@ -3,9 +3,9 @@ use regex::bytes::Regex; use std::str; use postgres; -use open; -use pkg; -use archive::{Archive,ArchiveEntry}; +use crate::open; +use crate::pkg; +use crate::archive::{Archive,ArchiveEntry}; fn getpkgsite(mut ent: Option<ArchiveEntry>) -> Result<ArchiveEntry> { @@ -19,7 +19,7 @@ fn getpkgsite(mut ent: Option<ArchiveEntry>) -> Result<ArchiveEntry> { } -pub fn sync(pg: &postgres::GenericConnection, sys: i32, mirror: &str) -> Result<()> { +pub fn sync<T: postgres::GenericClient>(pg: &mut T, sys: i32, mirror: &str) -> Result<()> { let path = format!("{}packagesite.txz", mirror); let mut rd = open::Path{path: &path, cache: true, canbelocal: false}.open()?; diff --git a/indexer/src/sys_rpm.rs b/indexer/src/sys_rpm.rs index 6e16916..bf5dca0 100644 --- a/indexer/src/sys_rpm.rs +++ b/indexer/src/sys_rpm.rs @@ -2,41 +2,25 @@ use std::collections::HashSet; use std::io::BufReader; use std::str::FromStr; use std::error::Error; -use std::fmt; use chrono::NaiveDateTime; use postgres; use quick_xml as xml; use quick_xml::events::Event; -use archive; -use open; -use pkg; -use man; +use crate::archive; +use crate::open; +use crate::pkg; +use crate::man; -// Ugh, quick-xml's Error type does not implement Error. -#[derive(Debug)] -struct XmlError(String); -impl fmt::Display for XmlError { - fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { write!(f, "{}", self.0) } -} -impl Error for XmlError { - fn description(&self) -> &str { self.0.as_ref() } -} -fn to_err(e: xml::Error) -> XmlError { - XmlError(format!("{}", e)) -} - - - -fn xml_getattr(e: &xml::events::BytesStart, attr: &str) -> Result<String,Box<Error>> { +fn xml_getattr(e: &xml::events::BytesStart, attr: &str) -> Result<String,Box<dyn Error>> { for kv in e.attributes().with_checks(false) { - let kv = kv.map_err(to_err)?; + let kv = kv?; if kv.key == attr.as_bytes() { return Ok(String::from_utf8(kv.value.into_owned())?); } } - Err(Box::new(XmlError(format!("Attribute '{}' not found", attr)))) + Err(Box::new(xml::Error::UnexpectedToken(format!("Attribute '{}' not found", attr)))) } @@ -53,7 +37,7 @@ struct PkgInfo { // Shared function to read primary.xml.gz and filelists.xml.gz. Runs the callback for each package // with the info that was found. -fn readpkgs<F>(url: String, mut cb: F) -> Result<(),Box<Error>> +fn readpkgs<F>(url: String, mut cb: F) -> Result<(),Box<dyn Error>> where F: FnMut(PkgInfo) { debug!("Reading {}", url); @@ -74,8 +58,7 @@ fn readpkgs<F>(url: String, mut cb: F) -> Result<(),Box<Error>> loop { buf.clear(); - let event = xml.read_event(&mut buf); - let event = event.map_err(to_err)?; + let event = xml.read_event(&mut buf)?; match event { @@ -97,7 +80,7 @@ fn readpkgs<F>(url: String, mut cb: F) -> Result<(),Box<Error>> Event::Text(e) => if savestr { - saved = Some(e.unescape_and_decode(&xml).map_err(to_err)?); + saved = Some(e.unescape_and_decode(&xml)?); savestr = false }, @@ -126,7 +109,7 @@ fn readpkgs<F>(url: String, mut cb: F) -> Result<(),Box<Error>> // Reads repomd.xml and returns the path to the primary.xml.gz and filelists.xml.gz -fn repomd(url: String) -> Result<(String,String),Box<Error>> { +fn repomd(url: String) -> Result<(String,String),Box<dyn Error>> { debug!("Reading {}", url); let mut fd = open::Path{path: &url, cache: true, canbelocal: false}.open()?; let mut xml = xml::Reader::from_reader( @@ -143,7 +126,7 @@ fn repomd(url: String) -> Result<(String,String),Box<Error>> { loop { buf.clear(); - let event = xml.read_event(&mut buf).map_err(to_err)?; + let event = xml.read_event(&mut buf)?; match event { Event::Start(ref e) | Event::Empty(ref e) => { @@ -173,7 +156,7 @@ fn repomd(url: String) -> Result<(String,String),Box<Error>> { } -pub fn sync(pg: &postgres::GenericConnection, sys: i32, cat: &str, mirror: &str) -> Result<(),Box<Error>> { +pub fn sync<T: postgres::GenericClient>(pg: &mut T, sys: i32, cat: &str, mirror: &str) -> Result<(),Box<dyn Error>> { let(primary, filelists) = repomd(format!("{}repodata/repomd.xml", mirror))?; let mut pkgswithman = HashSet::new(); diff --git a/indexer/src/sys_rpmdir.rs b/indexer/src/sys_rpmdir.rs index 730aea6..b922b43 100644 --- a/indexer/src/sys_rpmdir.rs +++ b/indexer/src/sys_rpmdir.rs @@ -2,10 +2,10 @@ use std::io::Result; use regex::Regex; use postgres; -use open; -use pkg; +use crate::open; +use crate::pkg; -pub fn sync(pg: &postgres::GenericConnection, sys: i32, cat: &str, mirror: &str) -> Result<()> { +pub fn sync<T: postgres::GenericClient>(pg: &mut T, sys: i32, cat: &str, mirror: &str) -> Result<()> { let pkgs : Vec<String> = open::Path{path: mirror, cache: true, canbelocal: false} .dirlist()?.into_iter() .filter_map(|(n,d)| if d { None } else { Some(n) }) diff --git a/lib/ManUtils/ManUtils.pm b/lib/ManUtils/ManUtils.pm index 8b157ca..251a20b 100644 --- a/lib/ManUtils/ManUtils.pm +++ b/lib/ManUtils/ManUtils.pm @@ -13,6 +13,33 @@ require XSLoader; XSLoader::load('ManUtils', $VERSION); +sub _groff { + my($input, $output, $errors, $cv, @cmd) = @_; + + # $MANWIDTH works by using the following groff options: -rLL=100n -rLT=100n + splice @cmd, 1, 0, qw|-Tutf8 -DUTF-8 -P-c -rLL=80n -rLT=80n|; + + $input = + # Disable hyphenation, since that screws up man page references. :-( + ".hy 0\n.de hy\n..\n" + # Emulate man-db's --nj option + .".na\n.de ad\n..\n" + .$input; + + my $groff = run_cmd \@cmd, + '<' => \$input, + '>' => \my $fmt, + '2>' => sub { if($_[0]) { chomp(my $e = $_[0]); push @$errors, "groff: $e" } }; + + $groff->cb(sub { + $$output = $fmt ? decode_utf8($fmt) : ''; + $$output =~ s/[\t\s\r\n]+$//; + $cv->send; + }); + $cv +} + + # Usage: $cv = fmt($input, \$output, \@errors) # $cv = AnyEvent condition variable, fired when done. # $input = UTF-8 encoded manual page source @@ -24,18 +51,13 @@ sub fmt { $$output = ''; @$errors = (); - $input = - # Disable hyphenation, since that screws up man page references. :-( - ".hy 0\n.de hy\n..\n" - # Emulate man-db's --nj option - .".na\n.de ad\n..\n" - .$input; - $input = encode_utf8($input); + # grog has a tendency to recognize pod2man generated pages as -ms, let's just work around that by enforcing -man + #return _groff $input, $output, $errors, $cv, 'groff', '-man' if $input =~ /^.\\" Automatically generated by Pod::Man/; + # Call grog to figure out which preprocessors to use. - # $MANWIDTH works by using the following groff options: -rLL=100n -rLT=100n - my $grog = run_cmd [qw|grog -Tutf8 -P-c -DUTF-8 -rLL=80n -rLT=80n -|], + my $grog = run_cmd [qw|grog -Tutf8 -DUTF-8 -|], '<' => \$input, '>' => \my $cmd, '2>' => sub { $_[0] && push @$errors, "grog: $_[0]" }; @@ -66,20 +88,9 @@ sub fmt { @cmd = (@cmd[0..$#cmd-1], $macros, $cmd[$#cmd]); push @$errors, "grog detected several macro packages: $double. Using $macros. (@cmd)"; } + @cmd = map $_ eq '-ms' ? '-man' : $_, @cmd; # -ms is almost(?) always wrong. - # grog 1.22.3 somehow loses the -P-c argument, let's make sure it's in the list. - splice @cmd, 1, 0, '-P-c'; - - my $groff = run_cmd \@cmd, - '<' => \$input, - '>' => \my $fmt, - '2>' => sub { if($_[0]) { chomp(my $e = $_[0]); push @$errors, "groff: $e" } }; - - $groff->cb(sub { - $$output = $fmt ? decode_utf8($fmt) : ''; - $$output =~ s/[\t\s\r\n]+$//; - $cv->send; - }); + _groff $input, $output, $errors, $cv, @cmd; }); $cv; diff --git a/schema.sql b/schema.sql new file mode 100644 index 0000000..e2c02c3 --- /dev/null +++ b/schema.sql @@ -0,0 +1,121 @@ +CREATE TABLE systems ( + -- Manually assigned number. The id is also used for ordering different + -- releases of the same system, as identified by 'name'. + id integer PRIMARY KEY, + name varchar NOT NULL, + release varchar, + short varchar NOT NULL +); + + +CREATE TABLE contents ( + id SERIAL PRIMARY KEY, + -- 'hash' is the SHA1 of the man page file after decompression but *before* + -- encoding conversion and removing 0-bytes. This means taking sha1(content) + -- may not necessary match the hash, and it's possible for the same content + -- to be in the database under multiple hashes (but I suspect that's rare). + hash bytea NOT NULL UNIQUE, + content text NOT NULL +); + + +-- Unique man page, as identified by name & section +CREATE TABLE mans ( + id SERIAL PRIMARY KEY, + name text NOT NULL, + section text NOT NULL, + UNIQUE(name, section) +); +CREATE INDEX mans_name ON mans USING btree(lower(name) text_pattern_ops); + + +-- List of man page locales for efficient referencing. Some locales include +-- the encoding in their name, which isn't really correct or even necessary +-- since we convert everything to UTF-8 anyway, but w/e, Can fix later. +CREATE TABLE locales ( + id SMALLSERIAL PRIMARY KEY, + locale text NOT NULL UNIQUE +); + + +-- List of encodings for efficient referencing. +CREATE TABLE encodings ( + id SMALLSERIAL PRIMARY KEY, + encoding text NOT NULL UNIQUE +); + + +CREATE TABLE packages ( + id SERIAL PRIMARY KEY, + system integer NOT NULL REFERENCES systems(id) ON DELETE CASCADE ON UPDATE CASCADE, + category varchar NOT NULL, + name varchar NOT NULL, + -- Whether this package has been seen in the last repository update. This + -- field is only updated for a few systems that are likely to delete packages + -- over time; non-rolling-release distros tend to not delete packages after + -- all. + -- Packages where the latest version does not have any man pages may also be + -- marked as dead even if the package is still available in the repos. + dead boolean NOT NULL DEFAULT FALSE, + UNIQUE(system, name, category) -- Note the order, lookups on (system,name) are common +); + + +CREATE TABLE package_versions ( + id SERIAL PRIMARY KEY, + package integer NOT NULL REFERENCES packages(id) ON DELETE CASCADE, + version varchar NOT NULL, + released date NOT NULL, + arch varchar, + UNIQUE(package, version) +); + + +CREATE TABLE files ( + pkgver integer NOT NULL REFERENCES package_versions(id) ON DELETE CASCADE, + man integer NOT NULL REFERENCES mans(id), + content integer NOT NULL REFERENCES content(id), + shorthash integer NOT NULL, -- cache: hash_to_shorthash(content.hash) + locale smallint NOT NULL REFERENCES locales(id) + -- The original encoding the man page was found in. This column isn't really + -- used at the moment, but is potentially useful when investigating encoding + -- issues. + encoding smallint NOT NULL REFERENCES encodings(id), + filename text NOT NULL, + PRIMARY KEY(pkgver, filename) +); +CREATE INDEX ON files (man, shorthash); +CREATE INDEX ON files (content); + + +-- For stats_cache +\i util/update_indices.sql + + + +-- Interpret first 4 bytes of hash as a signed 32-bit integer. +CREATE OR REPLACE FUNCTION hash_to_shorthash(hash bytea) RETURNS integer AS $$ + SELECT CASE WHEN get_byte(hash, 3) < 128 + THEN (get_byte(hash, 3)::int<<24) + (get_byte(hash, 2)::int<<16) + (get_byte(hash, 1)::int<<8) + get_byte(hash, 0) + ELSE -2147483648 + ((get_byte(hash, 3)::int - 128)<<24) + (get_byte(hash, 2)::int<<16) + (get_byte(hash, 1)::int<<8) + get_byte(hash, 0) + END; +$$ LANGUAGE SQL IMMUTABLE; + + +CREATE OR REPLACE FUNCTION is_english_locale(locale text) RETURNS bool AS $$ + SELECT locale IS NULL OR locale = '' OR locale LIKE 'en%'; +$$ IMMUTABLE LANGUAGE SQL; + + +CREATE OR REPLACE FUNCTION is_standard_man_location(path text) RETURNS bool AS $$ + SELECT path LIKE '/usr/share/man/man%' OR path LIKE '/usr/local/man/man%'; +$$ IMMUTABLE LANGUAGE sql; + +-- Convenient function to match the first character of a string. Second argument must be lowercase 'a'-'z' or '0'. +-- Postgres can inline and partially evaluate this function into the query plan, so it's fairly efficient. +CREATE OR REPLACE FUNCTION match_firstchar(str text, chr text) RETURNS boolean AS $$ + SELECT CASE WHEN chr = '0' + THEN (ascii(str) < 97 OR ascii(str) > 122) AND (ascii(str) < 65 OR ascii(str) > 90) + ELSE ascii(str) IN(ascii(chr),ascii(upper(chr))) + END; +$$ LANGUAGE SQL IMMUTABLE; diff --git a/sql/schema.sql b/sql/schema.sql deleted file mode 100644 index 23f7110..0000000 --- a/sql/schema.sql +++ /dev/null @@ -1,281 +0,0 @@ -CREATE TABLE systems ( - id integer PRIMARY KEY, -- hardcoded ID. - name varchar NOT NULL, - release varchar, - relorder integer NOT NULL DEFAULT 0, -- simple way of ordering different releases for the same system - short varchar NOT NULL -); - -CREATE TABLE contents ( - hash bytea PRIMARY KEY, - content varchar NOT NULL -); - -CREATE TABLE packages ( - id SERIAL PRIMARY KEY, - system integer NOT NULL REFERENCES systems(id) ON DELETE CASCADE, - category varchar, - name varchar NOT NULL, - UNIQUE(system, name, category) -- Note the order, lookups on (system,name) are common -); - -CREATE TABLE package_versions ( - id SERIAL PRIMARY KEY, - package integer NOT NULL REFERENCES packages(id) ON DELETE CASCADE, - version varchar NOT NULL, - released date NOT NULL, - arch varchar, - UNIQUE(package, version) -); - -CREATE TABLE man ( - package integer NOT NULL REFERENCES package_versions(id) ON DELETE CASCADE, - name varchar NOT NULL, - filename varchar NOT NULL, - locale varchar, - hash bytea NOT NULL REFERENCES contents(hash), - section varchar NOT NULL, - encoding varchar, - UNIQUE(package, filename) -); - -CREATE INDEX ON man (hash); -CREATE INDEX ON man (name); - - - -CREATE TABLE man_index AS SELECT DISTINCT name, section FROM man; -CREATE INDEX ON man_index USING btree(lower(name) text_pattern_ops); - -CREATE TABLE stats_cache AS SELECT count(distinct hash) AS hashes, count(distinct name) AS mans, count(*) AS files, count(distinct package) AS packages FROM man; - - - -INSERT INTO systems (id, name, release, short, relorder) VALUES - (1, 'Arch Linux', NULL, 'arch', 0), - (2, 'Ubuntu', '4.10', 'ubuntu-warty', 0), - (3, 'Ubuntu', '5.04', 'ubuntu-hoary', 1), - (4, 'Ubuntu', '5.10', 'ubuntu-breezy', 2), - (5, 'Ubuntu', '6.06', 'ubuntu-dapper', 3), - (6, 'Ubuntu', '6.10', 'ubuntu-edgy', 4), - (7, 'Ubuntu', '7.04', 'ubuntu-feisty', 5), - (8, 'Ubuntu', '7.10', 'ubuntu-gutsy', 6), - (9, 'Ubuntu', '8.04', 'ubuntu-hardy', 7), - (10, 'Ubuntu', '8.10', 'ubuntu-intrepid', 8), - (11, 'Ubuntu', '9.04', 'ubuntu-jaunty', 9), - (12, 'Ubuntu', '9.10', 'ubuntu-karmic', 10), - (13, 'Ubuntu', '10.04', 'ubuntu-lucid', 11), - (14, 'Ubuntu', '10.10', 'ubuntu-maverick', 12), - (15, 'Ubuntu', '11.04', 'ubuntu-natty', 13), - (16, 'Ubuntu', '11.10', 'ubuntu-oneiric', 14), - (17, 'Ubuntu', '12.04', 'ubuntu-precise', 15), - (18, 'Debian', '1.1', 'debian-buzz', 0), - (19, 'Debian', '1.2', 'debian-rex', 1), - (20, 'Debian', '1.3', 'debian-bo', 2), - (21, 'Debian', '2.0', 'debian-hamm', 3), - (22, 'Debian', '2.1', 'debian-slink', 4), - (23, 'Debian', '2.2', 'debian-potato', 5), - (24, 'Debian', '3.0', 'debian-woody', 6), - (25, 'Debian', '3.1', 'debian-sarge', 7), - (26, 'Debian', '4.0', 'debian-etch', 8), - (27, 'Debian', '5.0', 'debian-lenny', 9), - (28, 'Debian', '6.0', 'debian-squeeze', 10), - (29, 'FreeBSD', '1.0', 'freebsd-1.0', 0), - (30, 'FreeBSD', '2.0.5', 'freebsd-2.0.5', 1), - (31, 'FreeBSD', '2.1.5', 'freebsd-2.1.5', 2), - (32, 'FreeBSD', '2.1.7', 'freebsd-2.1.7', 3), - (33, 'FreeBSD', '2.2.2', 'freebsd-2.2.2', 4), - (34, 'FreeBSD', '2.2.5', 'freebsd-2.2.5', 5), - (35, 'FreeBSD', '2.2.6', 'freebsd-2.2.6', 6), - (36, 'FreeBSD', '2.2.7', 'freebsd-2.2.7', 7), - (37, 'FreeBSD', '2.2.8', 'freebsd-2.2.8', 8), - (38, 'FreeBSD', '3.0', 'freebsd-3.0', 9), - (39, 'FreeBSD', '3.1', 'freebsd-3.1', 10), - (40, 'FreeBSD', '3.2', 'freebsd-3.2', 11), - (41, 'FreeBSD', '3.3', 'freebsd-3.3', 12), - (42, 'FreeBSD', '3.4', 'freebsd-3.4', 13), - (43, 'FreeBSD', '3.5', 'freebsd-3.5', 14), - (44, 'FreeBSD', '3.5.1', 'freebsd-3.5.1', 15), - (45, 'FreeBSD', '4.0', 'freebsd-4.0', 16), - (46, 'FreeBSD', '4.1', 'freebsd-4.1', 17), - (47, 'FreeBSD', '4.1.1', 'freebsd-4.1.1', 18), - (48, 'FreeBSD', '4.2', 'freebsd-4.2', 19), - (49, 'FreeBSD', '4.3', 'freebsd-4.3', 20), - (50, 'FreeBSD', '4.4', 'freebsd-4.4', 21), - (51, 'FreeBSD', '4.5', 'freebsd-4.5', 22), - (52, 'FreeBSD', '4.6', 'freebsd-4.6', 23), - (53, 'FreeBSD', '4.6.2', 'freebsd-4.6.2', 24), - (54, 'FreeBSD', '4.7', 'freebsd-4.7', 25), - (55, 'FreeBSD', '4.8', 'freebsd-4.8', 26), - (56, 'FreeBSD', '4.9', 'freebsd-4.9', 27), - (57, 'FreeBSD', '4.10', 'freebsd-4.10', 28), - (58, 'FreeBSD', '4.11', 'freebsd-4.11', 29), - (59, 'FreeBSD', '5.0', 'freebsd-5.0', 30), - (60, 'FreeBSD', '5.1', 'freebsd-5.1', 31), - (61, 'FreeBSD', '5.2', 'freebsd-5.2', 32), - (62, 'FreeBSD', '5.2.1', 'freebsd-5.2.1', 33), - (63, 'FreeBSD', '5.3', 'freebsd-5.3', 34), - (64, 'FreeBSD', '5.4', 'freebsd-5.4', 35), - (65, 'FreeBSD', '5.5', 'freebsd-5.5', 36), - (66, 'FreeBSD', '6.0', 'freebsd-6.0', 37), - (67, 'FreeBSD', '6.1', 'freebsd-6.1', 38), - (68, 'FreeBSD', '6.2', 'freebsd-6.2', 39), - (69, 'FreeBSD', '6.3', 'freebsd-6.3', 40), - (70, 'FreeBSD', '6.4', 'freebsd-6.4', 41), - (71, 'FreeBSD', '7.0', 'freebsd-7.0', 42), - (72, 'FreeBSD', '7.1', 'freebsd-7.1', 43), - (73, 'FreeBSD', '7.2', 'freebsd-7.2', 44), - (74, 'FreeBSD', '7.3', 'freebsd-7.3', 45), - (75, 'FreeBSD', '7.4', 'freebsd-7.4', 46), - (76, 'FreeBSD', '8.0', 'freebsd-8.0', 47), - (77, 'FreeBSD', '8.1', 'freebsd-8.1', 48), - (78, 'FreeBSD', '8.2', 'freebsd-8.2', 49), - (79, 'FreeBSD', '8.3', 'freebsd-8.3', 50), - (80, 'FreeBSD', '9.0', 'freebsd-9.0', 52), - (81, 'Ubuntu', '12.10', 'ubuntu-quantal', 16), - (82, 'Ubuntu', '13.04', 'ubuntu-raring', 17), - (83, 'Debian', '7.0', 'debian-wheezy', 11), - (84, 'FreeBSD', '8.4', 'freebsd-8.4', 51), - (85, 'FreeBSD', '9.1', 'freebsd-9.1', 53), - (86, 'FreeBSD', '9.2', 'freebsd-9.2', 54), - (87, 'Ubuntu', '13.10', 'ubuntu-saucy', 18), - (88, 'Ubuntu', '14.04', 'ubuntu-trusty', 19), - (89, 'Ubuntu', '14.10', 'ubuntu-utopic', 20), - (90, 'Ubuntu', '15.04', 'ubuntu-vivid', 21), - (91, 'Debian', '8.0', 'debian-jessie', 12), - (92, 'Ubuntu', '15.10', 'ubuntu-wily', 22), - (93, 'Ubuntu', '16.04', 'ubuntu-xenial', 23), - (94, 'FreeBSD', '9.3', 'freebsd-9.3', 55), - (95, 'FreeBSD', '10.0', 'freebsd-10.0', 56), - (96, 'FreeBSD', '10.1', 'freebsd-10.1', 57), - (97, 'FreeBSD', '10.2', 'freebsd-10.2', 58), - (98, 'FreeBSD', '10.3', 'freebsd-10.3', 59), - (99, 'FreeBSD', '11.0', 'freebsd-11.0', 61), - (100,'Ubuntu', '16.10', 'ubuntu-yakkety', 24), - (101,'Fedora', '1', 'fedora-1', 0), - (102,'Fedora', '2', 'fedora-2', 1), - (103,'Fedora', '3', 'fedora-3', 2), - (104,'Fedora', '4', 'fedora-4', 3), - (105,'Fedora', '5', 'fedora-5', 4), - (106,'Fedora', '6', 'fedora-6', 5), - (107,'Fedora', '7', 'fedora-7', 6), - (108,'Fedora', '8', 'fedora-8', 7), - (109,'Fedora', '9', 'fedora-9', 8), - (110,'Fedora', '10', 'fedora-10', 9), - (111,'Fedora', '11', 'fedora-11', 10), - (112,'Fedora', '12', 'fedora-12', 11), - (113,'Fedora', '13', 'fedora-13', 12), - (114,'Fedora', '14', 'fedora-14', 13), - (115,'Fedora', '15', 'fedora-15', 14), - (116,'Fedora', '16', 'fedora-16', 15), - (117,'Fedora', '17', 'fedora-17', 16), - (118,'Fedora', '18', 'fedora-18', 17), - (119,'Fedora', '19', 'fedora-19', 18), - (120,'Fedora', '20', 'fedora-20', 19), - (121,'Fedora', '21', 'fedora-21', 20), - (122,'Fedora', '22', 'fedora-22', 21), - (123,'Fedora', '23', 'fedora-23', 22), - (124,'Fedora', '24', 'fedora-24', 23), - (125,'Fedora', '25', 'fedora-25', 24), - (126,'Ubuntu', '17.04', 'ubuntu-zesty', 25), - (127,'Debian', '9.0', 'debian-stretch', 13), - (128,'Fedora', '26', 'fedora-26', 25), - (129,'FreeBSD', '11.1', 'freebsd-11.1', 62), - (130,'Ubuntu', '17.10', 'ubuntu-artful', 26), - (131,'Fedora', '27', 'fedora-27', 26), - (132,'FreeBSD', '10.4', 'freebsd-10.4', 60), - (133,'CentOS', '2.1', 'centos-2.1', 1), - (134,'CentOS', '3.1', 'centos-3.1', 2), - (136,'CentOS', '3.3', 'centos-3.3', 4), - (137,'CentOS', '3.4', 'centos-3.4', 5), - (138,'CentOS', '3.5', 'centos-3.5', 6), - (139,'CentOS', '3.6', 'centos-3.6', 7), - (140,'CentOS', '3.7', 'centos-3.7', 8), - (141,'CentOS', '3.8', 'centos-3.8', 9), - (142,'CentOS', '3.9', 'centos-3.9', 10), - (143,'CentOS', '4.0', 'centos-4.0', 11), - (144,'CentOS', '4.1', 'centos-4.1', 12), - (145,'CentOS', '4.2', 'centos-4.2', 13), - (146,'CentOS', '4.3', 'centos-4.3', 14), - (147,'CentOS', '4.4', 'centos-4.4', 15), - (148,'CentOS', '4.5', 'centos-4.5', 16), - (149,'CentOS', '4.6', 'centos-4.6', 17), - (150,'CentOS', '4.7', 'centos-4.7', 18), - (151,'CentOS', '4.8', 'centos-4.8', 19), - (152,'CentOS', '4.9', 'centos-4.9', 20), - (153,'CentOS', '5.0', 'centos-5.0', 21), - (154,'CentOS', '5.1', 'centos-5.1', 22), - (155,'CentOS', '5.2', 'centos-5.2', 23), - (156,'CentOS', '5.3', 'centos-5.3', 24), - (157,'CentOS', '5.4', 'centos-5.4', 25), - (158,'CentOS', '5.5', 'centos-5.5', 26), - (159,'CentOS', '5.6', 'centos-5.6', 27), - (160,'CentOS', '5.7', 'centos-5.7', 28), - (161,'CentOS', '5.8', 'centos-5.8', 29), - (162,'CentOS', '5.9', 'centos-5.9', 30), - (163,'CentOS', '5.10', 'centos-5.10', 31), - (164,'CentOS', '5.11', 'centos-5.11', 32), - (165,'CentOS', '6.0', 'centos-6.0', 33), - (166,'CentOS', '6.1', 'centos-6.1', 34), - (167,'CentOS', '6.2', 'centos-6.2', 35), - (168,'CentOS', '6.3', 'centos-6.3', 36), - (169,'CentOS', '6.4', 'centos-6.4', 37), - (170,'CentOS', '6.5', 'centos-6.5', 38), - (171,'CentOS', '6.6', 'centos-6.6', 39), - (172,'CentOS', '6.7', 'centos-6.7', 40), - (173,'CentOS', '6.8', 'centos-6.8', 41), - (174,'CentOS', '6.9', 'centos-6.9', 42), - (175,'CentOS', '7.0', 'centos-7.0', 44), - (176,'CentOS', '7.1', 'centos-7.1', 45), - (177,'CentOS', '7.2', 'centos-7.2', 46), - (178,'CentOS', '7.3', 'centos-7.3', 47), - (179,'CentOS', '7.4', 'centos-7.4', 48), - (180,'Ubuntu', '18.04', 'ubuntu-bionic', 27), - (181,'Fedora', '28', 'fedora-28', 27), - (182,'CentOS', '7.5', 'centos-7.5', 49), - (183,'CentOS', '6.10', 'centos-6.10', 43), - (184,'FreeBSD', '11.2', 'freebsd-11.2', 63), - (185,'Ubuntu', '18.10', 'ubuntu-cosmic', 28), - (186,'Fedora', '29', 'fedora-29', 28), - (187,'CentOS', '7.6', 'centos-7.6', 50), - (188,'FreeBSD', '12.0', 'freebsd-12.0', 70), - (189,'Ubuntu', '19.04', 'ubuntu-disco', 29), - (190,'Debian', '10.0', 'debian-buster', 14), - (191,'Fedora', '30', 'fedora-30', 29); - - --- Removes any path components and compression extensions from the filename. -CREATE OR REPLACE FUNCTION basename_from_filename(fn text) RETURNS text AS $$ -DECLARE - ret text; - tmp text; -BEGIN - ret := regexp_replace(fn, '^.+/([^/]+)', E'\\1'); - LOOP - tmp := regexp_replace(regexp_replace(regexp_replace(ret, E'\\.gz$', ''), E'\\.lzma$', ''), E'\\.bz2$', ''); - EXIT WHEN tmp = ret; - ret := tmp; - END LOOP; - RETURN ret; -END; -$$ LANGUAGE plpgsql; - - -CREATE OR REPLACE FUNCTION section_from_filename(text) RETURNS text AS $$ - SELECT regexp_replace(basename_from_filename($1), E'^.+\\.([^.]+)$', E'\\1'); -$$ LANGUAGE SQL; - - -CREATE OR REPLACE FUNCTION name_from_filename(text) RETURNS text AS $$ - SELECT regexp_replace(basename_from_filename($1), E'^(.+)\\.[^.]+$', E'\\1'); -$$ LANGUAGE SQL; - - -CREATE OR REPLACE FUNCTION is_english_locale(locale text) RETURNS bool AS $$ - SELECT locale IS NULL OR locale LIKE 'en%'; -$$ IMMUTABLE LANGUAGE SQL; - - -CREATE OR REPLACE FUNCTION is_standard_man_location(path text) RETURNS bool AS $$ - SELECT path LIKE '/usr/share/man/man%' OR path LIKE '/usr/local/man/man%'; -$$ IMMUTABLE LANGUAGE sql; diff --git a/sql/update-2016-10-02.sql b/sql/update-2016-10-02.sql deleted file mode 100644 index a669314..0000000 --- a/sql/update-2016-10-02.sql +++ /dev/null @@ -1,30 +0,0 @@ -CREATE TABLE packages ( - id SERIAL PRIMARY KEY, - system integer NOT NULL REFERENCES systems(id), - category varchar, - name varchar NOT NULL, - UNIQUE(system, name, category) -- Note the order, lookups on (system,name) are common -); - -CREATE TABLE package_versions ( - id SERIAL PRIMARY KEY, - package integer NOT NULL REFERENCES packages(id), - version varchar NOT NULL, - released date NOT NULL, - UNIQUE(package, version) -); - -INSERT INTO packages (system, category, name) SELECT system, category, name FROM package GROUP BY system, category, name; -INSERT INTO package_versions (id, package, version, released) - SELECT p.id, pn.id, p.version, p.released FROM package p JOIN packages pn ON pn.system = p.system AND pn.category = p.category AND pn.name = p.name; - -SELECT setval('package_versions_id_seq', nextval('package_id_seq')); - -ALTER TABLE man DROP CONSTRAINT man_package_fkey; -ALTER TABLE man ADD FOREIGN KEY (package) REFERENCES package_versions(id); - --- Use a proper b-tree index -DROP INDEX man_hash_idx; -CREATE INDEX ON man (hash); - --- DROP TABLE package; diff --git a/sql/update-2016-10-03.sql b/sql/update-2016-10-03.sql deleted file mode 100644 index 68605d2..0000000 --- a/sql/update-2016-10-03.sql +++ /dev/null @@ -1,4 +0,0 @@ --- This check is not consistent with the HTML-check in util/add_dir.pl, but it --- happens to match exactly the same man pages currently. -DELETE FROM man WHERE section = 'html'; -DELETE FROM contents c WHERE NOT EXISTS(SELECT 1 FROM man m WHERE m.hash = c.hash); diff --git a/sql/update-2016-10-06.sql b/sql/update-2016-10-06.sql deleted file mode 100644 index ff4b394..0000000 --- a/sql/update-2016-10-06.sql +++ /dev/null @@ -1,3 +0,0 @@ -ALTER TABLE packages DROP CONSTRAINT packages_system_fkey, ADD CONSTRAINT packages_system_fkey FOREIGN KEY(system) REFERENCES systems(id) ON DELETE CASCADE; -ALTER TABLE package_versions DROP CONSTRAINT package_versions_package_fkey, ADD CONSTRAINT package_versions_package_fkey FOREIGN KEY(package) REFERENCES packages(id) ON DELETE CASCADE; -ALTER TABLE man DROP CONSTRAINT man_package_fkey, ADD CONSTRAINT man_package_fkey FOREIGN KEY(package) REFERENCES package_versions(id) ON DELETE CASCADE; diff --git a/sql/update-2016-10-09.sql b/sql/update-2016-10-09.sql deleted file mode 100644 index 3c3ed7c..0000000 --- a/sql/update-2016-10-09.sql +++ /dev/null @@ -1,7 +0,0 @@ -CREATE OR REPLACE FUNCTION is_english_locale(locale text) RETURNS bool AS $$ - SELECT locale IS NULL OR locale LIKE 'en%'; -$$ IMMUTABLE LANGUAGE SQL; - -CREATE OR REPLACE FUNCTION is_standard_man_location(path text) RETURNS bool AS $$ - SELECT path LIKE '/usr/share/man/man%' OR path LIKE '/usr/local/man/man%'; -$$ IMMUTABLE LANGUAGE sql; diff --git a/sql/update-2016-10-16.sql b/sql/update-2016-10-16.sql deleted file mode 100644 index 2a630a6..0000000 --- a/sql/update-2016-10-16.sql +++ /dev/null @@ -1,19 +0,0 @@ --- Various non-manpages -DELETE FROM man - WHERE filename ~ '/Makefile\.(in|am)$' - OR filename ~ '/\.cvsignore(\.gz)?$' - OR filename !~ '/[^/]*\.[^/]*$' - OR filename ~ '/man\.tmp$'; - --- Wrong locales, found with: --- SELECT DISTINCT Locale FROM man ORDER BY locale; -UPDATE man SET locale = NULL - WHERE locale = '5man' - OR locale = 'c' - OR locale ~ '^man.?$' - OR locale ~ '^Man-Part[12]$'; - --- Man page containing only a '$1'. Likely a build failure in earlier FreeBSD releases. -DELETE FROM man WHERE hash = '\x5ea7b8101325c704551852f70b652e0a2b0d7c12'; - -DELETE FROM contents c WHERE NOT EXISTS(SELECT 1 FROM man m WHERE m.hash = c.hash); diff --git a/sql/update-2016-11-06.sql b/sql/update-2016-11-06.sql deleted file mode 100644 index 7f4e820..0000000 --- a/sql/update-2016-11-06.sql +++ /dev/null @@ -1,2 +0,0 @@ -ALTER TABLE package_versions ADD COLUMN arch varchar; -ALTER TABLE man ADD COLUMN encoding varchar; diff --git a/sql/update-2016-12-18.sql b/sql/update-2016-12-18.sql deleted file mode 100644 index 479e114..0000000 --- a/sql/update-2016-12-18.sql +++ /dev/null @@ -1,26 +0,0 @@ --- Delete some non-man-pages, these are already blacklisted in indexer/man.rs:validate(). --- Found with: --- select hash from contents where content like e'\x7fELF%' or length(content) < 9 or content = e'timestamp\n' or content = e'.so man3/\n'; -DELETE FROM man WHERE hash IN( - '\x0c8b9d6f753e8d8ec9276bfe98e993a133847642', - '\x2c0f4624792234e2c289eec5b6bad1c699f84128', - '\x2d8b07a585e3f072aaa2c2b0ebac91d9039ccd54', - '\x4853d24dbb7d81e4782ef1bb1162c143f808dda1', - '\x6dfe263ccab32880795dff4479d990ade1daa839', - '\x72d08cf649f7a5cb0a9434d5b591878b2f7a0df9', - '\x816aff0cb37d6c9ccbfa48fffec83e39b37193fa', - '\x8af6f3e189ca29abb3a0ba51d7ef5e7e70451639', - '\x8e5113f6f47ce34e0437c2105441dbb70f01491a', - '\x92d754c27d4a6f851505be63aad6366857060f42', - '\x9312a9f378cc7750c4f473e3fdbd1d9b4aaf1efa', - '\x9f83db09859f909ce36b5aa97ec412c09ea27a76', - '\xadc83b19e793491b1c6ea0fd8b46cd9f32e592fc', - '\xb226262bf9f49e1098612ffdfc01680f7f305f70', - '\xc1a1a4edf60cfd417d52fc7bf1698bf0b6c814e6', - '\xda39a3ee5e6b4b0d3255bfef95601890afd80709', - '\xe04aededffd5ff6a3bc1a5294796ce1efa4dc68d', - '\xfe847f886e2d883a946282a552123dbba00f9596', - '\xffecacd94bd2bc4488db35a6b761ed430a65ac8f' -); - -DELETE FROM contents c WHERE NOT EXISTS(SELECT 1 FROM man m WHERE m.hash = c.hash); diff --git a/util/alpine.sh b/util/alpine.sh new file mode 100755 index 0000000..fa63e1f --- /dev/null +++ b/util/alpine.sh @@ -0,0 +1,105 @@ +#!/bin/sh + +. ./common.sh + +MIRROR=http://dl-cdn.alpinelinux.org/alpine/ + +alp() { + local VER=$1 + local REPOS=$2 + + for REPO in $REPOS; do + index alpine --sys alpine-$VER --mirror ${MIRROR}v$VER --repo $REPO + done +} + +case "$1" in + 3.0) + alp 3.0 main + ;; + 3.1) + alp 3.1 main + ;; + 3.2) + alp 3.2 main + ;; + 3.3) + alp 3.3 "main community" + ;; + 3.4) + alp 3.4 "main community" + ;; + 3.5) + alp 3.5 "main community" + ;; + 3.6) + alp 3.6 "main community" + ;; + 3.7) + alp 3.7 "main community" + ;; + 3.8) + alp 3.8 "main community" + ;; + 3.9) + alp 3.9 "main community" + ;; + 3.10) + alp 3.10 "main community" + ;; + 3.11) + alp 3.11 "main community" + ;; + 3.12) + alp 3.12 "main community" + ;; + 3.13) + alp 3.13 "main community" + ;; + 3.14) + alp 3.14 "main community" + ;; + 3.15) + alp 3.15 "main community" + ;; + 3.16) + alp 3.16 "main community" + ;; + 3.17) + alp 3.17 "main community" + ;; + 3.18) + alp 3.18 "main community" + ;; + 3.19) + alp 3.19 "main community" + ;; + old) + $0 3.0 + $0 3.1 + $0 3.2 + $0 3.3 + $0 3.4 + $0 3.5 + $0 3.6 + $0 3.7 + $0 3.8 + $0 3.9 + $0 3.10 + $0 3.11 + $0 3.12 + $0 3.13 + $0 3.14 + $0 3.15 + ;; + current) + $0 3.16 # till 2024-05-23 + $0 3.17 # till 2024-11-22 + $0 3.18 # till 2025-05-09 + $0 3.19 # till 2025-11-01 + ;; + all) + $0 old + $0 current + ;; +esac diff --git a/util/arch.sh b/util/arch.sh index 36e1b95..894bd91 100755 --- a/util/arch.sh +++ b/util/arch.sh @@ -2,7 +2,7 @@ . ./common.sh -MIRROR=http://mirror.i3d.net/pub/archlinux/ +MIRROR=http://mirror.ams1.nl.leaseweb.net/archlinux/ case "$1" in current) diff --git a/util/centos.sh b/util/centos.sh index ee879d2..5ad0541 100755 --- a/util/centos.sh +++ b/util/centos.sh @@ -4,6 +4,7 @@ VMIRROR=http://vault.centos.org/ CMIRROR=http://centos.mirrors.ovh.net/ftp.centos.org/ +SMIRROR=http://mirror.transip.net/centos-stream/ # Centos 3.1 - 3.6 (doesn't have useful repo metadata) centa() { @@ -38,6 +39,30 @@ centc() { index rpm --sys centos-$VER --cat centosplus --mirror "$MIR$DIR/centosplus/x86_64/" } +# CentOS 8.0+ +centd() { + local VER=$1 + local DIR=$2 + local MIR=${3:-$VMIRROR} + index rpm --sys centos-$VER --cat BaseOS --mirror "$MIR$DIR/BaseOS/x86_64/os/" + index rpm --sys centos-$VER --cat AppStream --mirror "$MIR$DIR/AppStream/x86_64/os/" + index rpm --sys centos-$VER --cat PowerTools --mirror "$MIR$DIR/PowerTools/x86_64/os/" + index rpm --sys centos-$VER --cat extras --mirror "$MIR$DIR/extras/x86_64/os/" + index rpm --sys centos-$VER --cat centosplus --mirror "$MIR$DIR/centosplus/x86_64/os/" +} + +# CentOS Stream 9+ +cente() { + local VER=$1 + local DIR=$2 + local MIR=${3:-$VMIRROR} + index rpm --sys centos-$VER --cat BaseOS --mirror "$MIR$DIR/BaseOS/x86_64/os/" + index rpm --sys centos-$VER --cat AppStream --mirror "$MIR$DIR/AppStream/x86_64/os/" + index rpm --sys centos-$VER --cat CRB --mirror "$MIR$DIR/CRB/x86_64/os/" + # There's also NFV and RT (with lots of overlap) and HighAvailability and ResilientStorage (with lots of overlap). + # Not sure which of those should be included. +} + case "$1" in 2.1) index rpmdir --sys centos-2.1 --cat core --mirror "${VMIRROR}2.1/final/i386/CentOS/RPMS/" @@ -186,6 +211,39 @@ case "$1" in 7.6) centc 7.6 7.6.1810 $CMIRROR ;; + 7.7) + centc 7.7 7.7.1908 $CMIRROR + ;; + 7.8) + centc 7.8 7.8.2003 $CMIRROR + ;; + 7.9) + centc 7.9 7.9.2009 $CMIRROR + ;; + 8.0) + centd 8.0 8.0.1905 $CMIRROR + ;; + 8.1) + centd 8.1 8.1.1911 $CMIRROR + ;; + 8.2) + centd 8.2 8.2.2004 $CMIRROR + ;; + 8.3) + centd 8.3 8.3.2011 $CMIRROR + ;; + 8.4) + centd 8.4 8.4.2105 $CMIRROR + ;; + 8.5) + centd 8.5 8.5.2111 $CMIRROR + ;; + 8-stream) + centd 8-stream 8-stream $CMIRROR + ;; + 9-stream) + cente 9-stream 9-stream $SMIRROR + ;; old) $0 2.1 $0 3.1 @@ -235,10 +293,20 @@ case "$1" in $0 7.3 $0 7.4 $0 7.5 + $0 7.6 + $0 7.7 + $0 7.8 + $0 8.0 + $0 8.1 + $0 8.2 + $0 8.3 + $0 8.4 + $0 8.5 ;; current) - $0 6.10 # till 2020-11-30 - $0 7.6 # till 2024-06-30 + $0 7.9 # till 2024-06-30 + $0 8-stream + $0 9-stream ;; all) $0 old diff --git a/util/cron.sh b/util/cron.sh index 237f7d3..1da2321 100755 --- a/util/cron.sh +++ b/util/cron.sh @@ -3,15 +3,12 @@ PSQL="psql -U manned -Awtq" +./alpine.sh current ./arch.sh current ./debian.sh current ./centos.sh current ./fedora.sh current ./ubuntu.sh current -# Only update indices once a week (on mondays). This process is slow and the data doesn't often change anyway. -if [ `date +%u` == 1 ] -then - echo "============ Updating SQL indices" - $PSQL -f update_indices.sql -fi +echo "============ Updating SQL indices" +$PSQL -f update_indices.sql diff --git a/util/debian.sh b/util/debian.sh index 52c9dd5..c91772c 100755 --- a/util/debian.sh +++ b/util/debian.sh @@ -63,6 +63,18 @@ case "$1" in index_deb debian-buster $CMIRROR buster "main contrib non-free" cmp amd64 index_deb debian-buster $CMIRROR buster-updates "main contrib non-free" cmp amd64 ;; + bullseye) + index_deb debian-bullseye $CMIRROR bullseye "main contrib non-free" cmp amd64 + index_deb debian-bullseye $CMIRROR bullseye "main contrib non-free" cmp all + ;; + bookworm) + index_deb debian-bookworm $CMIRROR bookworm "main contrib non-free" cmp amd64 + index_deb debian-bookworm $CMIRROR bookworm "main contrib non-free" cmp all + ;; + trixie) + index_deb debian-trixie $CMIRROR trixie "main contrib non-free" cmp amd64 + index_deb debian-trixie $CMIRROR trixie "main contrib non-free" cmp all + ;; old) $0 buzz $0 rex @@ -77,10 +89,13 @@ case "$1" in $0 squeeze $0 wheezy $0 jessie + $0 stretch ;; current) - $0 stretch $0 buster + $0 bullseye + $0 bookworm + $0 trixie ;; all) $0 old diff --git a/util/fedora.sh b/util/fedora.sh index 8670f05..1d7df15 100755 --- a/util/fedora.sh +++ b/util/fedora.sh @@ -113,6 +113,42 @@ case "$1" in index rpm --sys fedora-30 --cat everything --mirror "${CMIRROR}releases/30/Everything/x86_64/os/" index rpm --sys fedora-30 --cat everything --mirror "${CMIRROR}updates/30/Everything/x86_64/" ;; + 31) + index rpm --sys fedora-31 --cat everything --mirror "${CMIRROR}releases/31/Everything/x86_64/os/" + index rpm --sys fedora-31 --cat everything --mirror "${CMIRROR}updates/31/Everything/x86_64/" + ;; + 32) + index rpm --sys fedora-32 --cat everything --mirror "${CMIRROR}releases/32/Everything/x86_64/os/" + index rpm --sys fedora-32 --cat everything --mirror "${CMIRROR}updates/32/Everything/x86_64/" + ;; + 33) + index rpm --sys fedora-33 --cat everything --mirror "${CMIRROR}releases/33/Everything/x86_64/os/" + index rpm --sys fedora-33 --cat everything --mirror "${CMIRROR}updates/33/Everything/x86_64/" + ;; + 34) + index rpm --sys fedora-34 --cat everything --mirror "${CMIRROR}releases/34/Everything/x86_64/os/" + index rpm --sys fedora-34 --cat everything --mirror "${CMIRROR}updates/34/Everything/x86_64/" + ;; + 35) + index rpm --sys fedora-35 --cat everything --mirror "${CMIRROR}releases/35/Everything/x86_64/os/" + index rpm --sys fedora-35 --cat everything --mirror "${CMIRROR}updates/35/Everything/x86_64/" + ;; + 36) + index rpm --sys fedora-36 --cat everything --mirror "${CMIRROR}releases/36/Everything/x86_64/os/" + index rpm --sys fedora-36 --cat everything --mirror "${CMIRROR}updates/36/Everything/x86_64/" + ;; + 37) + index rpm --sys fedora-37 --cat everything --mirror "${CMIRROR}releases/37/Everything/x86_64/os/" + index rpm --sys fedora-37 --cat everything --mirror "${CMIRROR}updates/37/Everything/x86_64/" + ;; + 38) + index rpm --sys fedora-38 --cat everything --mirror "${CMIRROR}releases/38/Everything/x86_64/os/" + index rpm --sys fedora-38 --cat everything --mirror "${CMIRROR}updates/38/Everything/x86_64/" + ;; + 39) + index rpm --sys fedora-39 --cat everything --mirror "${CMIRROR}releases/39/Everything/x86_64/os/" + index rpm --sys fedora-39 --cat everything --mirror "${CMIRROR}updates/39/Everything/x86_64/" + ;; old) $0 1 $0 2 @@ -142,10 +178,19 @@ case "$1" in $0 26 $0 27 $0 28 - ;; - current) $0 29 $0 30 + $0 31 + $0 32 + $0 33 + $0 34 + $0 35 + $0 36 + ;; + current) + $0 37 + $0 38 + $0 39 ;; all) $0 old diff --git a/util/freebsd.sh b/util/freebsd.sh index 8eb56aa..285c9a3 100755 --- a/util/freebsd.sh +++ b/util/freebsd.sh @@ -583,12 +583,78 @@ case $1 in index_core freebsd-11.2 "${MIR}base.txz" core-base 2018-06-27 index freebsd2 --sys freebsd-11.2 --mirror "$PKG" ;; + 11.3) + MIR="${CMIRROR}amd64/11.3-RELEASE/" + PKG="${PMIRROR}FreeBSD:11:amd64/release_3/" + index_core freebsd-11.3 "${MIR}base.txz" core-base 2019-07-09 + index freebsd2 --sys freebsd-11.3 --mirror "$PKG" + ;; + 11.4) + MIR="${CMIRROR}amd64/11.4-RELEASE/" + PKG="${PMIRROR}FreeBSD:11:amd64/release_4/" + index_core freebsd-11.4 "${MIR}base.txz" core-base 2020-06-23 + index freebsd2 --sys freebsd-11.4 --mirror "$PKG" + ;; 12.0) MIR="${CMIRROR}amd64/12.0-RELEASE/" PKG="${PMIRROR}FreeBSD:11:amd64/release_0/" index_core freebsd-12.0 "${MIR}base.txz" core-base 2018-12-11 index freebsd2 --sys freebsd-12.0 --mirror "$PKG" ;; + 12.1) + MIR="${CMIRROR}amd64/12.1-RELEASE/" + PKG="${PMIRROR}FreeBSD:12:amd64/release_1/" + index_core freebsd-12.1 "${MIR}base.txz" core-base 2019-11-04 + index freebsd2 --sys freebsd-12.1 --mirror "$PKG" + ;; + 12.2) + MIR="${CMIRROR}amd64/12.2-RELEASE/" + PKG="${PMIRROR}FreeBSD:12:amd64/release_2/" + index_core freebsd-12.2 "${MIR}base.txz" core-base 2020-10-27 + index freebsd2 --sys freebsd-12.2 --mirror "$PKG" + ;; + 12.3) + MIR="${CMIRROR}amd64/12.3-RELEASE/" + PKG="${PMIRROR}FreeBSD:12:amd64/release_3/" + index_core freebsd-12.3 "${MIR}base.txz" core-base 2021-12-07 + index freebsd2 --sys freebsd-12.3 --mirror "$PKG" + ;; + 12.4) + MIR="${CMIRROR}amd64/12.4-RELEASE/" + PKG="${PMIRROR}FreeBSD:12:amd64/release_4/" + index_core freebsd-12.4 "${MIR}base.txz" core-base 2022-12-05 + index freebsd2 --sys freebsd-12.4 --mirror "$PKG" + ;; + 13.0) + MIR="${CMIRROR}amd64/13.0-RELEASE/" + PKG="${PMIRROR}FreeBSD:13:amd64/release_0/" + index_core freebsd-13.0 "${MIR}base.txz" core-base 2021-04-13 + index freebsd2 --sys freebsd-13.0 --mirror "$PKG" + ;; + 13.1) + MIR="${CMIRROR}amd64/13.1-RELEASE/" + PKG="${PMIRROR}FreeBSD:13:amd64/release_1/" + index_core freebsd-13.1 "${MIR}base.txz" core-base 2022-05-16 + index freebsd2 --sys freebsd-13.1 --mirror "$PKG" + ;; + 13.2) + MIR="${CMIRROR}amd64/13.2-RELEASE/" + PKG="${PMIRROR}FreeBSD:13:amd64/release_2/" + index_core freebsd-13.2 "${MIR}base.txz" core-base 2023-04-11 + index freebsd2 --sys freebsd-13.2 --mirror "$PKG" + ;; + 13.3) + MIR="${CMIRROR}amd64/13.3-RELEASE/" + PKG="${PMIRROR}FreeBSD:13:amd64/release_3/" + index_core freebsd-13.3 "${MIR}base.txz" core-base 2024-03-05 + index freebsd2 --sys freebsd-13.3 --mirror "$PKG" + ;; + 14.0) + MIR="${CMIRROR}amd64/14.0-RELEASE/" + PKG="${PMIRROR}FreeBSD:14:amd64/release_0/" + index_core freebsd-14.0 "${MIR}base.txz" core-base 2023-11-20 + index freebsd2 --sys freebsd-14.0 --mirror "$PKG" + ;; old) $0 1.0 $0 2.0.5 @@ -654,6 +720,17 @@ case $1 in $0 11.0 $0 11.1 $0 11.2 + $0 11.3 + $0 11.4 $0 12.0 + $0 12.1 + $0 12.2 + $0 12.3 + $0 12.4 + $0 13.0 + $0 13.1 + $0 13.2 + $0 13.3 + $0 14.0 ;; esac diff --git a/util/netbsd.sh b/util/netbsd.sh new file mode 100755 index 0000000..3153e58 --- /dev/null +++ b/util/netbsd.sh @@ -0,0 +1,346 @@ +#!/bin/sh + +. ./common.sh + +CMIRROR=http://cdn.netbsd.org/pub/NetBSD/ +AMIRROR=http://archive.netbsd.org/pub/NetBSD-archive/ + +# NetBSD's distribution format has been delightfully stable across the years. +# This directory structure and the exact list of sets has been pretty much the +# same since 1.3. +index_base() { + local ver=$1 + local date=$2 + local ext=${3:-tar.xz} + local mirror=${4:-$CMIRROR} + local arch=${5:-amd64} + for pkg in comp games man text xbase xcomp xserver; do + index pkg --sys netbsd-$ver --cat base --pkg $pkg --ver $ver --date $date "${mirror}NetBSD-$ver/$arch/binary/sets/$pkg.$ext" + done +} + +case $1 in + 1.3) + index_base 1.3 1998-01-03 tgz $AMIRROR i386 + ;; + + 1.3.2) + index_base 1.3.2 1998-05-29 tgz $AMIRROR i386 + ;; + + 1.3.3) + index_base 1.3.3 1998-12-23 tgz $AMIRROR i386 + ;; + + 1.4) + index_base 1.4 1999-05-12 tgz $AMIRROR i386 + ;; + + 1.4.1) + index_base 1.4.1 1999-08-26 tgz $AMIRROR i386 + ;; + + 1.4.2) + index_base 1.4.2 2000-03-21 tgz $AMIRROR i386 + ;; + + 1.4.3) + index_base 1.4.3 2000-11-25 tgz $AMIRROR i386 + ;; + + 1.5) + index_base 1.5 2000-12-06 tgz $AMIRROR i386 + ;; + + 1.5.1) + index_base 1.5.1 2001-07-11 tgz $AMIRROR i386 + ;; + + 1.5.2) + index_base 1.5.2 2001-09-13 tgz $AMIRROR i386 + ;; + + 1.5.3) + index_base 1.5.3 2002-07-22 tgz $AMIRROR i386 + ;; + + 1.6) + index_base 1.6 2002-09-14 tgz $AMIRROR i386 + ;; + + 1.6.1) + index_base 1.6.1 2003-04-21 tgz $AMIRROR i386 + ;; + + 1.6.2) + index_base 1.6.2 2004-03-01 tgz $AMIRROR i386 + ;; + + 2.0) + index_base 2.0 2004-12-09 tgz $AMIRROR i386 + ;; + + 2.0.2) + index_base 2.0.2 2005-04-14 tgz $AMIRROR i386 + ;; + + # Archive has no binary files for this release :( + #2.0.3) + # index_base 2.0.3 2005-10-31 tgz $AMIRROR i386 + # ;; + + 2.1) + index_base 2.1 2005-11-02 tgz $AMIRROR i386 + ;; + + 3.0) + index_base 3.0 2005-12-23 tgz $AMIRROR i386 + ;; + + 3.0.1) + index_base 3.0.1 2006-07-24 tgz $AMIRROR i386 + ;; + + 3.0.2) + index_base 3.0.2 2006-11-04 tgz $AMIRROR i386 + ;; + + 3.1) + index_base 3.1 2006-11-04 tgz $AMIRROR i386 + ;; + + 4.0) + index_base 4.0 2007-12-19 tgz $AMIRROR i386 + ;; + + 4.0.1) + index_base 4.0.1 2008-10-14 tgz $AMIRROR i386 + ;; + + 5.0) + index_base 5.0 2009-04-29 tgz $AMIRROR i386 + ;; + + 5.0.1) + index_base 5.0.1 2009-08-02 tgz $AMIRROR i386 + ;; + + 5.0.2) + index_base 5.0.2 2010-02-12 tgz $AMIRROR i386 + ;; + + 5.1) + index_base 5.1 2010-11-19 tgz $AMIRROR i386 + ;; + + 5.1.2) + index_base 5.1.2 2012-02-02 tgz $AMIRROR i386 + ;; + + 5.1.3) + index_base 5.1.3 2012-09-28 tgz $AMIRROR i386 + ;; + + 5.1.4) + index_base 5.1.4 2014-01-27 tgz $AMIRROR i386 + ;; + + 5.1.5) + index_base 5.1.5 2014-11-15 tgz $AMIRROR i386 + ;; + + 5.2) + index_base 5.2 2012-12-03 tgz $AMIRROR i386 + ;; + + 5.2.1) + index_base 5.2.1 2012-09-28 tgz $AMIRROR i386 + ;; + + 5.2.2) + index_base 5.2.2 2014-01-27 tgz $AMIRROR i386 + ;; + + 5.2.3) + index_base 5.2.3 2014-11-15 tgz $AMIRROR i386 + ;; + + 6.0) + index_base 6.0 2012-10-17 tgz $AMIRROR + ;; + + 6.0.1) + index_base 6.0.1 2012-12-26 tgz $AMIRROR + ;; + + 6.0.2) + index_base 6.0.2 2013-05-18 tgz $AMIRROR + ;; + + 6.0.3) + index_base 6.0.3 2013-09-30 tgz $AMIRROR + ;; + + 6.0.4) + index_base 6.0.4 2014-01-27 tgz $AMIRROR + ;; + + 6.0.5) + index_base 6.0.5 2014-04-12 tgz $AMIRROR + ;; + + 6.0.6) + index_base 6.0.6 2014-09-22 tgz $AMIRROR + ;; + + 6.1) + index_base 6.1 2013-05-18 tgz $AMIRROR + ;; + + 6.1.1) + index_base 6.1.1 2013-08-15 tgz $AMIRROR + ;; + + 6.1.2) + index_base 6.1.2 2013-09-26 tgz $AMIRROR + ;; + + 6.1.3) + index_base 6.1.3 2014-01-18 tgz $AMIRROR + ;; + + 6.1.4) + index_base 6.1.4 2014-04-12 tgz $AMIRROR + ;; + + 6.1.5) + index_base 6.1.5 2014-09-22 tgz $AMIRROR + ;; + + 7.0) + index_base 7.0 2015-09-25 tgz $AMIRROR + ;; + + 7.0.1) + index_base 7.0.1 2016-05-22 tgz $AMIRROR + ;; + + 7.0.2) + index_base 7.0.2 2016-10-21 tgz $AMIRROR + ;; + + 7.1) + index_base 7.1 2017-03-11 tgz $AMIRROR + ;; + + 7.1.1) + index_base 7.1.1 2017-12-22 tgz $AMIRROR + ;; + + 7.1.2) + index_base 7.1.2 2018-03-15 tgz $AMIRROR + ;; + + 7.2) + index_base 7.2 2018-08-29 tgz $AMIRROR + ;; + + 8.0) + index_base 8.0 2018-07-17 tgz $AMIRROR + ;; + + 8.1) + index_base 8.1 2019-05-31 tgz $AMIRROR + ;; + + 8.2) + index_base 8.2 2020-03-31 tgz $AMIRROR + ;; + + 9.0) + index_base 9.0 2020-02-14 + ;; + + 9.1) + index_base 9.1 2020-10-18 + ;; + + 9.2) + index_base 9.2 2021-05-12 + ;; + + 9.3) + index_base 9.3 2022-08-04 + ;; + + 10.0) + index_base 10.0 2024-03-28 + ;; + + old) + $0 1.3 + $0 1.3.2 + $0 1.3.3 + $0 1.4 + $0 1.4.1 + $0 1.4.2 + $0 1.4.3 + $0 1.5 + $0 1.5.1 + $0 1.5.2 + $0 1.5.3 + $0 1.6 + $0 1.6.1 + $0 1.6.2 + $0 2.0 + $0 2.0.2 + #$0 2.0.3 + $0 2.1 + $0 3.0 + $0 3.0.1 + $0 3.0.2 + $0 3.1 + $0 4.0 + $0 4.0.1 + $0 5.0 + $0 5.0.1 + $0 5.0.2 + $0 5.1 + $0 5.1.2 + $0 5.1.3 + $0 5.1.4 + $0 5.1.5 + $0 5.2 + $0 5.2.1 + $0 5.2.2 + $0 5.2.3 + $0 6.0 + $0 6.0.1 + $0 6.0.2 + $0 6.0.3 + $0 6.0.4 + $0 6.0.5 + $0 6.0.6 + $0 6.1 + $0 6.1.1 + $0 6.1.2 + $0 6.1.3 + $0 6.1.4 + $0 6.1.5 + $0 7.0 + $0 7.0.1 + $0 7.0.2 + $0 7.1 + $0 7.1.1 + $0 7.1.2 + $0 7.2 + $0 8.0 + $0 8.1 + $0 8.2 + $0 9.0 + $0 9.1 + $0 9.2 + $0 9.3 + $0 10.0 + ;; +esac diff --git a/util/ubuntu.sh b/util/ubuntu.sh index bdd5e8f..0a7ba6f 100755 --- a/util/ubuntu.sh +++ b/util/ubuntu.sh @@ -3,7 +3,7 @@ . ./common.sh AMIRROR=http://old-releases.ubuntu.com/ubuntu/ -CMIRROR=http://nl.archive.ubuntu.com/ubuntu/ +CMIRROR=http://ftp.nluug.nl/os/Linux/distr/ubuntu/ # Shortcut for a standard Ubuntu repo, usage: @@ -116,6 +116,33 @@ case $1 in disco) stdrepo disco $CMIRROR amd64 ;; + eoan) + stdrepo eoan $CMIRROR amd64 + ;; + focal) + stdrepo focal $CMIRROR amd64 + ;; + groovy) + stdrepo groovy $CMIRROR amd64 + ;; + hirsute) + stdrepo hirsute $CMIRROR amd64 + ;; + impish) + stdrepo impish $CMIRROR amd64 + ;; + jammy) + stdrepo jammy $CMIRROR amd64 + ;; + kinetic) + stdrepo kinetic $CMIRROR amd64 + ;; + lunar) + stdrepo lunar $CMIRROR amd64 + ;; + mantic) + stdrepo mantic $CMIRROR amd64 + ;; old) $0 warty $0 hoary @@ -136,19 +163,28 @@ case $1 in $0 raring $0 quantal $0 saucy + $0 trusty $0 utopic $0 vivid $0 wily + $0 xenial $0 yakkety $0 zesty $0 artful + $0 cosmic + $0 disco + $0 eoan + $0 groovy + $0 hirsute + $0 impish + $0 kinetic ;; current) - $0 trusty # until 2019-04 - $0 xenial # until 2021-04 - $0 bionic # until 2023-04 - $0 cosmic # until 2019-07 - $0 disco # until 2020-01 + $0 bionic # until 2028-04 + $0 focal # until 2030-04 + $0 jammy # until 2032-04 + $0 lunar # until 2024-01 + $0 mantic # until 2024-07 ;; all) $0 old diff --git a/util/update_indices.sql b/util/update_indices.sql index a785ab7..b977d8a 100644 --- a/util/update_indices.sql +++ b/util/update_indices.sql @@ -2,14 +2,10 @@ -- the table being replaced. The site should remain responsive while these -- queries are run. BEGIN; -CREATE TABLE man_index_new AS SELECT DISTINCT name, section FROM man; -CREATE INDEX ON man_index_new USING btree(lower(name) text_pattern_ops); -DROP TABLE man_index; -ALTER TABLE man_index_new RENAME TO man_index; -COMMIT; - -BEGIN; -CREATE TABLE stats_cache_new AS SELECT count(distinct hash) AS hashes, count(distinct name) AS mans, count(*) AS files, count(distinct package) AS packages FROM man; +CREATE TABLE stats_cache_new AS + SELECT (SELECT count(*) FROM contents) AS hashes, + (SELECT count(distinct name) FROM mans) AS mans, * + FROM (SELECT count(*), count(distinct pkgver) FROM files) x(files, packages); DROP TABLE stats_cache; ALTER TABLE stats_cache_new RENAME TO stats_cache; COMMIT; diff --git a/web/Cargo.lock b/web/Cargo.lock index 1727e7b..0871c27 100644 --- a/web/Cargo.lock +++ b/web/Cargo.lock @@ -2,74 +2,46 @@ # It is not intended for manual editing. [[package]] name = "aho-corasick" -version = "0.7.3" +version = "0.7.18" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e37cfd5e7657ada45f742d6e99ca5788580b5c529dc78faf11ece6dc702656f" dependencies = [ - "memchr 2.2.0 (registry+https://github.com/rust-lang/crates.io-index)", + "memchr", ] [[package]] name = "lazy_static" -version = "1.3.0" +version = "1.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e2abad23fbc42b3700f2f279844dc832adb2b2eb069b2df918f455c4e18cc646" [[package]] name = "memchr" -version = "2.2.0" +version = "2.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "308cc39be01b73d0d18f82a0e7b2a3df85245f84af96fdddc5d202d27e47b86a" [[package]] name = "regex" -version = "1.1.6" +version = "1.5.4" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d07a8629359eb56f1e2fb1652bb04212c072a87ba68546a04065d525673ac461" dependencies = [ - "aho-corasick 0.7.3 (registry+https://github.com/rust-lang/crates.io-index)", - "memchr 2.2.0 (registry+https://github.com/rust-lang/crates.io-index)", - "regex-syntax 0.6.6 (registry+https://github.com/rust-lang/crates.io-index)", - "thread_local 0.3.6 (registry+https://github.com/rust-lang/crates.io-index)", - "utf8-ranges 1.0.2 (registry+https://github.com/rust-lang/crates.io-index)", + "aho-corasick", + "memchr", + "regex-syntax", ] [[package]] name = "regex-syntax" -version = "0.6.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "ucd-util 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "thread_local" -version = "0.3.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "lazy_static 1.3.0 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "ucd-util" -version = "0.1.3" -source = "registry+https://github.com/rust-lang/crates.io-index" - -[[package]] -name = "utf8-ranges" -version = "1.0.2" +version = "0.6.25" source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f497285884f3fcff424ffc933e56d7cbca511def0c9831a7f9b5f6153e3cc89b" [[package]] name = "web" version = "0.1.0" dependencies = [ - "lazy_static 1.3.0 (registry+https://github.com/rust-lang/crates.io-index)", - "regex 1.1.6 (registry+https://github.com/rust-lang/crates.io-index)", + "lazy_static", + "regex", ] - -[metadata] -"checksum aho-corasick 0.7.3 (registry+https://github.com/rust-lang/crates.io-index)" = "e6f484ae0c99fec2e858eb6134949117399f222608d84cadb3f58c1f97c2364c" -"checksum lazy_static 1.3.0 (registry+https://github.com/rust-lang/crates.io-index)" = "bc5729f27f159ddd61f4df6228e827e86643d4d3e7c32183cb30a1c08f604a14" -"checksum memchr 2.2.0 (registry+https://github.com/rust-lang/crates.io-index)" = "2efc7bc57c883d4a4d6e3246905283d8dae951bb3bd32f49d6ef297f546e1c39" -"checksum regex 1.1.6 (registry+https://github.com/rust-lang/crates.io-index)" = "8f0a0bcab2fd7d1d7c54fa9eae6f43eddeb9ce2e7352f8518a814a4f65d60c58" -"checksum regex-syntax 0.6.6 (registry+https://github.com/rust-lang/crates.io-index)" = "dcfd8681eebe297b81d98498869d4aae052137651ad7b96822f09ceb690d0a96" -"checksum thread_local 0.3.6 (registry+https://github.com/rust-lang/crates.io-index)" = "c6b53e329000edc2b34dbe8545fd20e55a333362d0a321909685a19bd28c3f1b" -"checksum ucd-util 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)" = "535c204ee4d8434478593480b8f86ab45ec9aae0e83c568ca81abf0fd0e88f86" -"checksum utf8-ranges 1.0.2 (registry+https://github.com/rust-lang/crates.io-index)" = "796f7e48bef87609f7ade7e06495a87d5cd06c7866e6a5cbfceffc558a243737" diff --git a/www/images/alpine.png b/www/images/alpine.png Binary files differnew file mode 100644 index 0000000..ccc839b --- /dev/null +++ b/www/images/alpine.png diff --git a/www/images/gradients.png b/www/images/gradients.png Binary files differdeleted file mode 100644 index c6cb96d..0000000 --- a/www/images/gradients.png +++ /dev/null diff --git a/www/images/netbsd.png b/www/images/netbsd.png Binary files differnew file mode 100644 index 0000000..c4a78d0 --- /dev/null +++ b/www/images/netbsd.png diff --git a/www/images/search.png b/www/images/search.png Binary files differdeleted file mode 100644 index c1bd5cf..0000000 --- a/www/images/search.png +++ /dev/null diff --git a/www/index.pl b/www/index.pl index 0df8980..fbec00f 100755 --- a/www/index.pl +++ b/www/index.pl @@ -1,1135 +1,1177 @@ #!/usr/bin/perl -use strict; +use v5.26; use warnings; -use TUWF ':html', 'html_escape', ':xml'; -use JSON::XS; +use TUWF ':html5_', ':xml'; use POSIX 'ceil'; +use SQL::Interp 'sql', 'sql_interp'; +use Time::Local 'timegm'; use Cwd 'abs_path'; our $ROOT; BEGIN { ($ROOT = abs_path $0) =~ s{/www/index\.pl$}{}; } +# Force the pure-perl AnyEvent backend; More lightweight and we don't need the +# performance of EV. Fixes an issue with subprocess spawning under TUWF's +# built-in web server that I haven't been able to track down. +BEGIN { $ENV{PERL_ANYEVENT_MODEL} = 'Perl'; } use lib "$ROOT/lib/ManUtils/inst/lib/perl5"; use ManUtils; TUWF::set( - logfile => $ENV{TUWF_LOG}, - db_login => [undef, undef, undef], - debug => 0, - xml_pretty => 0, - log_slow_pages => 500, - # Cache the system information - pre_request_handler => sub { - my $self = shift; - if(!$self->{systems}) { - $self->{systems} = $self->dbSystemGet; - $_->{full} = $_->{name}.($_->{release}?' '.$_->{release}:'') for(@{$self->{systems}}); - $self->{sysbyid} = { map +($_->{id}, $_), @{$self->{systems}} }; - $self->{sysbyshort} = { map +($_->{short}, $_), @{$self->{systems}} }; - } - 1; - }, - error_404_handler => sub { - my $self = shift; - $self->resStatus(404); - my $title = 'No manual entry for '.$self->reqPath; - $self->htmlHeader(title => $title); - h1 $title; - p 'That is, the page you were looking for doesn\'t exist.'; - $self->htmlFooter; - }, + logfile => $ENV{TUWF_LOG}, + db_login => [undef, undef, undef], + debug => $ENV{TUWF_DEBUG}, + xml_pretty => 0, + log_slow_pages => 500, ); -TUWF::register( - qr// => \&home, - qr{info/about} => \&about, - qr{browse/search} => \&browsesearch, - - # These have to go before the other mappings, to ensure that links work for - # man pages called 'pkg' or 'man'. This also means that we can't have a - # system named 8 hex digits, but at least that's easy to guarantee. :) - qr{([^/]+)/([0-9a-f]{8})} => \&man, - qr{([^/]+)/([0-9a-f]{8})/src} => \&src, - # We don't have any other single-component paths - qr{([^/]+)} => \&man, - - qr{pkg/([^/]+)} => \&pkg_list, - # pkg/$system/$category/$name (/$version); $category may contain a slash, too. - qr{pkg/([^/]+)/(.+)} => \&pkg_info, - - # Redirects for canonical URLs - qr{man/([^/]+)/(.+)} => \&man_redir, - - # Redirects for old URLs. - # /browse/<pkg> has been moved to /pkg/ with the package category added to the path - qr{browse/([^/]+)} => sub { $_[0]->resRedirect("/pkg/$_[1]", 'perm'); }, - qr{browse/([^/]+)/([^/]+)(?:/([^/]+))?} => sub { - my($self, $sys, $name, $ver) = @_; - $sys = $self->{sysbyshort}{$sys}; - return $self->resNotFound if !$sys; - my $pkgs = $self->dbPackageGet(sysid => $sys->{id}, name => $name, results => 1); - return $self->resNotFound if !@$pkgs; - $self->resRedirect("/pkg/$sys->{short}/$pkgs->[0]{category}/$name".($ver ? "/$ver" :''), 'perm'); - }, - - # Redirect for a specific language for a man page. - # I'm not a fan of this solution; might drop it in the future. - qr{lang/([^/]+)/([^/]+)} => sub { - my($s, $l, $n) = @_; - $n = _normalizename($n); - my($m, undef) = $s->dbManPrefName($n, language => $l); - return $s->resNotFound if !$m; - $s->resRedirect("/$m->{name}/".substr($m->{hash}, 0, 8), 'temp'); - }, - - qr{xml/search\.xml} => \&xmlsearch, - qr{json/tree\.json} => \&jsontree, -); +TUWF::hook before => sub { + if(tuwf->{_TUWF}{http}) { + if(tuwf->resFile("$ROOT/www", tuwf->reqPath)) { + tuwf->resHeader('Cache-Control' => 'max-age=31536000'); + tuwf->done; + } + } +}; -TUWF::run(); +# TODO: Add SQL::Interp support to TUWF directly, in some form. +sub TUWF::Object::dbExeci { shift->dbExec(sql_interp @_) } +sub TUWF::Object::dbVali { shift->dbVal (sql_interp @_) } +sub TUWF::Object::dbRowi { shift->dbRow (sql_interp @_) } +sub TUWF::Object::dbAlli { shift->dbAll (sql_interp @_) } +sub TUWF::Object::dbPagei { shift->dbPage(shift, sql_interp @_) } -sub home { - my $self = shift; - my $stats = $self->dbStats; - my $fn = sub { local $_=shift; 1 while(s/(\d)(\d{3})($|,)/$1,$2/); $_ }; - $self->htmlHeader(title => 'Man Pages Archive'); - h1 'Man Pages Archive'; - p class => 'txt'; lit sprintf <<' _', map $fn->($stats->{$_}), qw|hashes mans files packages|; - Indexing <b>%s</b> versions of <b>%s</b> manual pages found in <b>%s</b> - files of <b>%s</b> packages. - <br /><br /> - Manned.org aims to index all manual pages from a variety of systems, both - old and new, and provides a convenient interface for looking up and viewing - the various versions of each man page. - <a href="/info/about">About manned.org »</a> - _ - end; - - h2 'Browse the manuals'; - ul id => 'systems'; - my %sys; - push @{$sys{$_->{name}}}, $_ for(@{$self->{systems}}); - for my $sys (sort keys %sys) { - $sys = $sys{$sys}; - (my $img = $sys->[0]{short}) =~ s/^(.+)-.+$/$1/; - li; - a href => "/pkg/$sys->[0]{short}" if @$sys == 1; - span style => "background-image: url('images/$img.png')", ''; - b $sys->[0]{name}; - if(@$sys > 1) { - my $i = 0; - for(reverse @$sys) { - a href => "/pkg/$_->{short}", ++$i > 3 ? (class => 'hidden') : (), $_->{release}; - lit ' '; - } - a href => "#", class => 'more', 'more...' if $i > 3; - } - end 'a' if @$sys == 1; - end; - } - end; - - h2 'Other sites'; - ul id => 'external'; - li; a href => 'http://man7.org/linux/man-pages/index.html', 'man7.org'; txt ' - Linux man pages from several upstream projects.'; end; - li; a href => 'http://manpag.es/', 'ManPag.es'; txt ' - Man pages from several Linux distributions.'; end; - li; a href => 'https://www.mankier.com/', 'ManKier'; txt ' - Fedora Rawhide + some manually imported man pages; Nicely formatted and with some unique features.'; end; - li; a href => 'http://man.cx/', 'man.cx'; txt ' - Man pages extracted from Debian testing.'; end; - li; a href => 'http://man.he.net/', 'man.he.net'; txt ' - Also seems to be from a Debian-like system.'; end; - li; a href => 'http://linux.die.net/man/', 'die.net'; txt ' - Seems to be based on an RPM-based Linux distribution.'; end; - li; a href => 'http://manpages.org/', 'manpages.org'; txt ' - Lots of mostly-nicely formatted man pages, no clue about source.'; end; - li; a href => 'http://www.manpagez.com/', 'manpagez.com'; txt ' - Mac OS X, has some GTK-html and texinfo documentation as well.'; end; - li; a href => 'https://jlk.fjfi.cvut.cz/arch/manpages/dev', 'Arch Linux Man Pages'; end; - li; a href => 'https://manpages.debian.org/', 'Debian Man Pages'; end; - li; a href => 'https://www.dragonflybsd.org/cgi/web-man', 'DragonFlyBSD Man Pages'; end; - li; a href => 'http://www.freebsd.org/cgi/man.cgi', 'FreeBSD.org Man Pages'; end; - li; a href => 'http://netbsd.gw.com/cgi-bin/man-cgi', 'NetBSD Man Pages'; end; - li; a href => 'http://www.openbsd.org/cgi-bin/man.cgi', 'OpenBSD Man Pages'; end; - li; a href => 'http://manpages.ubuntu.com/', 'Ubuntu Manuals'; end; - li; a href => 'https://man.voidlinux.eu/', 'Void Linux manpages'; end; - end; - $self->htmlFooter; +# Set the last modification time from a string in yyyy-mm-dd format. +sub TUWF::Object::resLastMod { + my($s, $d) = @_; + return if $d !~ /^(\d{4})-(\d{2})-(\d{2})/; + my @t = gmtime timegm 0,0,0,$3,$2-1,$1; + $s->resHeader('Last-Modified', sprintf '%s, %02d %s %04d %02d:%02d:%02d GMT', + (qw|Sun Mon Tue Wed Thu Fri Sat|)[$t[6]], $t[3], + (qw|Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec|)[$t[4]], + $t[5]+1900, $t[2], $t[1], $t[0]); } -sub about { - my $self = shift; - $self->htmlHeader(title => 'About'); - h1 'About Manned.org'; - div id => 'about'; - - h2 'Goal'; - p; lit <<' _'; - The state of online indices of manual pages used to be a sad one. Existing - sites used to only offer you a single version of a man page: From one - origin, and often only in a single language. Most didn't even tell you where - the manual actually originated from, making it very hard to determine - whether the manual you found actually applied to your situation and even - harder to find a manual for a specific system. Additionally, some sites - rendered the manuals in an unreadable way, didn't correctly handle special - formatting - like tables - or didn't correctly display non-ASCII characters. - <br /><br /> - Nowadays there are many good alternatives, but Manned.org was one of the - sites created in order to improve situation. This site aims to index the - manual pages from a variaty of systems, both old and new, and allows you to - browse through the various versions of a manual page to find out how each - system behaves. The manuals are stored in the database as UTF-8, and are - passed through <a href="http://www.gnu.org/software/groff/">groff</a> to - render them in (mostly) the same way as they are displayed in your terminal. - <br /><br /> - This website is <a href="https://code.blicky.net/yorhel/manned">open - source</a> (MIT licensed) and written in a combination of Perl and Rust. The - entire PostgreSQL database is available for download (see "Database - download" below). - _ - end; - - h2 'URL format'; - lit <<' _'; - <p>You can link to specific packages and man pages with several URL formats. - These URLs will keep working in the future, so you should not have to worry - about eventual dead links.</p> - <h3>Man pages</h3> - <p>The following URLs are available to refer to an individual man page:</p> - <dl> - <dt><code>/<name>/<8-hex-digits></code></dt><dd> - This is the permalink format for a specific man page (e.g. <a href="/ls/910be0ed">/ls/910be0ed</a>).</dd> - <dt><code>/<name>[.<section>]</code></dt><dd> - Will try to get the latest and most-close-to-upstream version of a man - page (e.g. <a href="/socket">/socket</a> or <a - href="/socket.7">/socket.7</a>). Note that this may fetch the man page - from any available system, so may result in confusing scenarios for - system-specific documentation.</dd> - <dt><code>/man/<system>/<name>[.<section>]</code></dt><dd> - Will get the latest version of a man page from the given system (e.g. <a - href="/man/ubuntu-xenial/rsync">/man/ubuntu-xenial/rsync</a>)</dd> - <dt><code>/man/<system>/<category>/<package>/<name>[.<section>]</code></dt><dd> - Will get the latest version of a man page from the given package (e.g. <a - href="/man/ubuntu-xenial/net/rsync/rsync">/man/ubuntu-xenial/net/rsync/rsync</a>)</dd> - <dt><code>/man/<system>/<category>/<package>/<version>/<name>[.<section>]</code></dt><dd> - Will get the man page from a specific package version (e.g. <a - href="/man/ubuntu-xenial/net/rsync/3.1.1-3ubuntu1/rsync">/man/ubuntu-xenial/net/rsync/3.1.1-3ubuntu1/rsync</a>)</dd> - </dl> - <p>Currently, the last three URLs will perform a redirect to the - appropriate permalink URL, but this may change in the future.<br /> - In all URLs where an optional <code>.<section></code> can be provided, - the search is performed as a prefix match. For example, <a - href="/cat.3">/cat.3</a> will provide the <code>cat.3tcl</code> man page if - no exact <code>cat.3</code> version is available. Linking to the full - section name is also possible: <a href="/cat.3tcl">/cat.3tcl</a>. If no - section is given and multiple sections are available, the lowest section - number is chosen.</p> - <h3>Packages</h3> - <p>Linking to individual packages is also possible. These pages will show a - listing of all manual pages available in the given package.</p> - <dl> - <dt><code>/pkg/<system>/<category>/<package></code></dt><dd> - For the latest version of a package (e.g. <a - href="/pkg/arch/core/coreutils">/pkg/arch/core/coreutils</a>).</dd> - <dt><code>/pkg/<system>/<category>/<package>/<version></code></dt><dd> - For a particular version of a package (e.g. <a - href="/pkg/arch/core/coreutils/8.25-2">/pkg/arch/core/coreutils/8.25-2</a>).</dd> - </dl> - <p>Note that this site only indexes packages that actually have manual - pages; Linking to a package that doesn't have any will result in a 404 - page.</p> - _ - - h2 'The indexing process'; - p; lit <<' _'; - All man pages are fetched right from the (binary) packages available on the - public repositories of Linux distributions. In particular:<br /> - <dl> - <dt>Arch Linux</dt><dd> - The core, extra and community repositories are fetched from a local - Arch mirror. Indexing started around begin June 2012. The i686 - architecture was indexed until November 6th, 2016, packages after that - were fetched from from x86_64.</dd> - <dt>Debian</dt><dd> - Historical releases were fetched from <a - href="http://archive.debian.org/debian/">http://archive.debian.org/debian/</a> - and <a href="http://snapshot.debian.org/">http://snapshot.debian.org/</a>. - For buzz, rex and bo, we're missing a few man pages because some packages - were missing from the repository archives. Where available, all components - (main, contrib and non-free) from the $release and $release-updates - repositories are indexed.</dd> - <dt>CentOS</dt><dd> - Historical releases were fetched from <a - href="http://vault.centos.org/">vault.centos.org</a>, current releases - from a local mirror. Where applicable, the following repositories were - indexed: addons, centosplus, contrib, extras, os. The i386 architecture - was indexed for versions lower than 7.0, since 7.0 the packages from - x86_64 are indexed. - <dt>Fedora</dt><dd> - Historical releases were fetched from <a - href="http://archives.fedoraproject.org/pub/archive/fedora/linux/">archives.fedoraproject.org</a>, - current releases from a local repository. Fedora Core 1 till 6 are - (incorrectly) called 'Fedora' here. To compensate for that, Fedora 3 till - 6 also include the Extras repository. For Fedora 7 and later, the - 'Everything' and 'updates' repositories are indexed. The i386 arch was - indexed for Fedora 17 and older, the x86_64 arch starting with Fedora - 18.</dd> - <dt>FreeBSD</dt><dd> - Historical releases were fetched from <a - href="http://ftp-archive.freebsd.org/mirror/FreeBSD-Archive/">http://ftp-archive.freebsd.org/mirror/FreeBSD-Archive/</a>. - The base installation tarballs are included in the database as packages - prefixed with <i>core-</i>. The package repositories have also been - indexed, except for 2.0.5 - 2.2.7 and 3.0 - 3.3 because those were not - available on the ftp archive. Only the -RELEASE repositories have been - included, which is generally a snapshot of the ports directory around the - time of the release. The release dates indicated for many packages were - guessed from the file modification dates in the tarball, and may be - inaccurate. The i368 arch was indexed for FreeBSD 11.0 and older, the - amd64 arch starting with 11.1.</dd> - <dt>Ubuntu</dt><dd> - Historical releases were fetched from <a - href="http://old-releases.ubuntu.com/ubuntu/">http://old-releases.ubuntu.com/ubuntu/</a>, - supported releases from a local mirror. All components (main, universe, - restricted and multiverse) from the $release, $release-updates and - $release-security repositories are indexed. Indexing started around mid - June 2012. All releases before 2017 were indexed from the i386 - repositories, starting with 17.04 the amd64 repositories were used.</dd> - </dl> - <p> - Only packages for a single architecture (i386 or amd64) are scanned. To my - knowledge, packages that come with different manuals for different - architectures either don't exist or are extremely rare. It does happen that - some packages are not available for all architectures. Usually, though, - every package is at least available for the most popular architecture, so - hopefully we're not missing out on much. <br /><br /> - The repositories are scanned for new packages on a daily basis. - </p> - _ - end; - - h2 'Database download'; - p; lit <<' _'; - This site is backed by a PostgreSQL database containing all the man pages. - Weekly dumps of the full database are available for download at - <a href="http://dl.manned.org/dumps/">http://dl.manned.org/dumps/</a>. - <br /><br /> - Be warned that the download server may not be terribly reliable, so it is - advisable to use a client that supports resumption of partial downloads. See - <a href="/wget">wget's -c</a> or <a href="/curl">curl's -C</a>. - <br /><br /> - The database schema is "documented" at <a - href="https://code.blicky.net/yorhel/manned/src/branch/master/sql/schema.sql">schema.sql</a> - in the git repo. Note that these dumps don't constitute a stable API and, - while this won't happen frequently, incompatible schema changes or Postgres - major version bumps may occur. - _ - end; - - h2 'Other systems'; - p; lit <<' _'; - Suggestions for new (or old) systems to index are welcome. - <br /><br /> - It would be great to index a few more non-Linux systems such as other BSDs, - Solaris/Illumos and Mac OS X. Unfortunately, those don't always follow a - binary package based approach, or are otherwise less easy to properly index. - <br /><br /> - In general, systems that follow an entirely source-based distribution - approach can't be indexed without compiling everything. Since that is both - very resource-heavy and open to security issues, there are no plans to - include manuals from such systems at the moment. So unless someone comes - with a solution I hadn't thought of yet, there won't be any Gentoo manuals - here. :-( - _ - end; - - h2 'Future plans'; - p; lit <<' _'; - This site isn't nearly as awesome yet as it could be. Here's some ideas that - would be nice to have in the future: - <ul> - <li>Improved, more intelligent, search,</li> - <li><a href="/apropos.1">apropos(1)</a> emulation(?),</li> - <li>Diffs between various versions of a man page,</li> - <li>Anchor links within man pages, for easier linking to a section or paragraph,</li> - <li>Alternative formats (Text, PDF, more semantic HTML, etc),</li> - <li>A command-line client, like <a href="/man.1">man(1)</a> with manned.org as database backend.</li> - </ul> - _ - end; - - h2 'Copyright'; - p; lit <<' _'; - All manual pages are copyrighted by their respective authors. The manuals - have been fetched from publically available repositories of free and - (primarily) open source software. The distributors of said software have put - in efforts to only include software and documentation that allows free - distribution. Nonetheless, if a manual that does not allow to be - redistributed has been inadvertently included in our index, please let me - know and I will have it removed as soon as possible. - _ - end; - - end; - $self->htmlFooter; +# The systems table doesn't change often, so keep an in-memory cache for quick lookups. +sub systems { + state $s ||= [ map { + $_->{full} = $_->{name}.($_->{release}?' '.$_->{release}:''); + $_ + } tuwf->dbAll('SELECT id, name, release, short FROM systems ORDER BY name, id')->@* ]; } +sub sysbyid { state $s ||= { map +($_->{id}, $_), systems->@* } } +sub sysbyshort { state $s ||= { map +($_->{short}, $_), systems->@* } } -sub paginate { - my($url, $count, $perpage, $p) = @_; - return if $count <= $perpage; - - my $l = sub { - my $c = shift; - a href => sprintf('%s%d', $url, $c), $c if $c != $p; - b $c if $c == $p; - }; - - my $lp = ceil($count/$perpage); - p class => 'paginate'; - $l->(1) if $p > 1+4; - b '...' if $p > 1+5; - $l->($_) for (($p > 4 ? $p-4 : 1)..($p+4 > $lp ? $lp : $p+4)); - b '...' if $p < $lp-5; - $l->($lp) if $p < $lp-4; - end; -} +# URL-unescape some special characters that may occur in man names. +# Firefox seems to escape [ and ] in URLs. It doesn't really have to... +sub normalize_name { $_[0] =~ s/%5b/[/irg =~ s/%5d/]/irg =~ s/%20/ /rg } +sub shorthash_to_hex { unpack 'H*', pack 'i', $_[0] } # int -> hex +sub shorthash_to_int { unpack 'i', pack 'H*', $_[0] } # hex -> int -sub browsesearch { - my $self = shift; - my $q = $self->reqGet('q')||''; - my $man = $self->dbSearch($q, 150); - - return $self->resRedirect("/$man->[0]{name}.$man->[0]{section}", 'temp') if @$man == 1; - - $self->htmlHeader(title => 'Search results for '.$q); - h1 'Search results for '.$q; - p 'Note: This is just a simple case-insensitive prefix match on the man names. In the future we\'ll have more powerful search functionality. Hopefully.'; - if(@$man) { - ul id => 'searchres'; - for(@$man) { - li; - a href => "/$_->{name}.$_->{section}", $_->{name}; - i " $_->{section}"; - end; - } - end; - } else { - br; br; - b 'No results :-('; - } - - $self->htmlFooter; -} +# Subquery returning all packages that have a man page. +my $packages_with_man = '(SELECT * FROM packages p WHERE EXISTS(SELECT 1 FROM package_versions pv WHERE pv.package = p.id AND EXISTS(SELECT 1 FROM files f WHERE f.pkgver = pv.id)))'; +sub escape_like { $_[0] =~ s/([_%\\])/\\$1/rg } -sub pkg_list { - my($self, $short) = @_; +sub sql_join { + my $sep = shift; + my @args = map +($sep, $_), @_; + sql @args[1..$#args]; +} +sub sql_and { @_ ? sql_join 'AND', map sql('(', $_, ')'), @_ : sql '1=1' } +sub sql_or { @_ ? sql_join 'OR', map sql('(', $_, ')'), @_ : sql '1=0' } - my $sys = $self->{sysbyshort}{$short}; - return $self->resNotFound if !$sys; - my $f = $self->formValidate( - { get => 'c', required => 0, enum => [ '0', 'all', 'a'..'z' ], default => 'all' }, - { get => 'p', required => 0, default => 1, template => 'uint', min => 1, max => 200 }, - ); - return $self->resNotFound if $f->{_err}; +sub pkg_frompath { + my($sys_where, $path) = @_; - my %opt = (hasman => 1, sysid => $sys->{id}, char => $f->{c} eq 'all' ? undef : $f->{c}); - my $pkg = $self->dbPackageGet(%opt, results => 200, page => $f->{p}); - my $count = $self->dbPackageGet(%opt, countonly => 1)->[0]{count}; + # $path should be "$category/$name" or "$category/$name/$version", since + # $category may contain a slash, let's try both options. - my $title = "Packages for $sys->{name}".($sys->{release}?" $sys->{release}":""); - $self->htmlHeader(title => $title); - div id => 'pkglist'; - h1 $title; + my sub lookup { + my($cat, $name) = @_; + tuwf->dbRowi('SELECT id, system, name, category FROM', $packages_with_man, 'p WHERE', $sys_where, 'AND category =', \$cat, 'AND name =', \$name); + } - p class => 'charselect'; - for('all', 0, 'a'..'z') { - a href => "/pkg/$short?c=$_", $_?uc$_:'#' if $_ ne $f->{c}; - b $_?uc$_:'#' if $_ eq $f->{c}; + # $category/$name + # e.g. contrib/games/alien + if($path =~ m{^(.+)/([^/]+)$}) { + my $pkg = lookup $1, $2; + return ($pkg, '') if $pkg->{id}; } - end; - - p 'Note: Packages without man pages are not listed.'; - paginate "/pkg/$short?c=$f->{c};p=", $count, 200, $f->{p}; - ul id => 'packages'; - for(@$pkg) { - li; - a href => "/pkg/$short/$_->{category}/$_->{name}", $_->{name}; - i ' '.$_->{category}; - end; + + # $category/$name/$version + # e.g. contrib/games/alien/10.2 + if($path =~ m{^(.+)/([^/]+)/([^/]+)$}) { + my $pkg = lookup $1, $2; + return ($pkg, $3) if $pkg->{id}; } - end; - paginate "/pkg/$short?c=$f->{c};p=", $count, 200, $f->{p}; - end; - $self->htmlFooter; + (undef, ''); } -sub pkg_frompath { - my($self, $sys, $path) = @_; - - # $path should be "$category/$name" or "$category/$name/$version", since - # $category may contain a slash, let's try both options. - - # $category/$name - # e.g. contrib/games/alien - if($path =~ m{^(.+)/([^/]+)$}) { - my($category, $name) = ($1, $2); - my $pkg = $self->dbPackageGet(sysid => $sys, category => $category, name => $name, hasman => 1)->[0]; - return ($pkg, '') if $pkg; - } - - # $category/$name/$version - # e.g. contrib/games/alien/10.2 - if($path =~ m{^(.+)/([^/]+)/([^/]+)$}) { - my($category, $name, $version) = ($1, $2, $3); - my $pkg = $self->dbPackageGet(sysid => $sys, category => $category, name => $name, hasman => 1)->[0]; - return ($pkg, $version) if $pkg; - } - - (undef, ''); +# Get the preferred man page for the given filters. +sub man_pref { + my($section, $where) = @_; + $where = sql_and $where, sql 'm.section LIKE', \(escape_like($section).'%') if length $section; + + # Criteria to determine a "preferred" man page: + # 1. english: English versions of a man page have preference over other locales + # 2. pkgver: Newer versions of the same package have preference over older versions + # 3. stdloc: Prefer man pages in standard locations + # 4. secmatch: Prefer an exact section match + # 5. arch: Prefer Arch over other systems (because it tends to be the most up-to-date, and closest to upstreams) + # 6. debian: If there's no Arch, prefer latest Debian over other systems (again, tends to be more up-to-date) + # (also resolves distro-specific tooling disputes such as https://code.blicky.net/yorhel/manned/issues/1 ) + # 7. sysrel: Prefer a more recent system release over an older release + # 8. secorder: Lower sections before higher sections (because man does it this way, for some reason) + # 9. pkgdate: Prefer more recent packages (cross-distro) + # 10. Fall back on shorthash comparison, to ensure the result is stable + + state $archid = sysbyshort->{arch}{id}; + state $debid = (sort { $b->{id} <=> $a->{id} } grep $_->{short} =~ /^debian-/, systems->@*)[0]{id}; + + tuwf->dbRowi(q{ + WITH unfiltered AS ( + SELECT m.name, m.section, l.locale, f.shorthash, f.content, f.filename, s AS sys, p AS pkg, v AS ver + FROM files f + JOIN locales l ON l.id = f.locale + JOIN mans m ON m.id = f.man + JOIN package_versions v ON v.id = f.pkgver + JOIN packages p ON p.id = v.package + JOIN systems s ON s.id = p.system + WHERE}, $where, q{ + ), f_english AS( + SELECT * FROM unfiltered WHERE NOT EXISTS(SELECT 1 FROM unfiltered WHERE is_english_locale(locale)) OR is_english_locale(locale) + ), f_pkgver AS( + SELECT * FROM f_english a WHERE NOT EXISTS(SELECT 1 FROM f_english b WHERE (a.ver).package = (b.ver).package AND (a.ver).released < (b.ver).released) + ), f_stdloc AS( + SELECT * FROM f_pkgver WHERE NOT EXISTS(SELECT 1 FROM f_pkgver WHERE is_standard_man_location(filename)) OR is_standard_man_location(filename) + ), f_secmatch AS( + SELECT * FROM f_stdloc WHERE NOT EXISTS(SELECT 1 FROM f_stdloc WHERE section =}, \$section, q{) OR section =}, \$section, q{ + ), f_arch AS( + SELECT * FROM f_secmatch WHERE NOT EXISTS(SELECT 1 FROM}, length $section ? 'f_secmatch' : 'f_stdloc', qq{WHERE (sys).id = $archid) OR (sys).id = $archid + ), f_debian AS( + SELECT * FROM f_arch WHERE NOT EXISTS(SELECT 1 FROM f_arch WHERE (sys).id = $debid) OR (sys).id = $debid + ), f_sysrel AS( + SELECT * FROM f_debian a WHERE NOT EXISTS(SELECT 1 FROM f_debian b WHERE (a.sys).name = (b.sys).name AND (a.sys).id < (b.sys).id) + ), f_secorder AS( + SELECT * FROM f_sysrel a WHERE NOT EXISTS(SELECT 1 FROM f_sysrel b WHERE a.section > b.section) + ), f_pkgdate AS( + SELECT * FROM f_secorder a WHERE NOT EXISTS(SELECT 1 FROM f_secorder b WHERE (a.ver).released < (b.ver).released) + ) + SELECT (pkg).system, (pkg).category, (pkg).name AS package, (ver).version, (ver).released, (ver).id AS verid, + name, section, filename, locale, shorthash, content + FROM f_pkgdate ORDER BY shorthash LIMIT 1 + }); } -sub pkg_info { - my($self, $short, $path) = @_; - - my $sys = $self->{sysbyshort}{$short}; - return $self->resNotFound if !$sys; - - my($pkg, $ver) = pkg_frompath($self, $sys->{id}, $path); - return $self->resNotFound if !$pkg; +# Given the name of a man page with optional section, find out the actual name +# and section suffix of the man page and the preferred version. +sub man_pref_name { + my($name, $where) = @_; + + # Check the <name>.<section> format first, because ~most~ cases where + # there's a collision in the format, the <name>-only page is either + # uninteresting or a file name parsing error. + if ($name =~ /^(.+)\.([^.]+)$/) { + my($n, $s) = ($1,$2); + my $man = man_pref $s, sql_and $where, sql 'm.name =', \$n; + return ($man, $s) if length $man->{name}; + } - my $vers = $self->dbPackageVersions($pkg->{id}); + my $man = man_pref undef, sql_and $where, sql 'm.name =', \$name; + length $man->{name} ? ($man, '') : (undef, ''); +} - my $sel = $ver ? (grep $_->{version} eq $ver, @$vers)[0] : $vers->[0]; - return $self->resNotFound if !$sel; - my $f = $self->formValidate({ get => 'p', required => 0, default => 1, template => 'uint', min => 1, max => 100}); - return $self->resNotFound if $f->{_err}; +sub framework_ { + my $content = pop; + my(%o) = @_; + + html_ lang => 'en', sub { + head_ sub { + link_ rel => 'stylesheet', type => 'text/css', href => '/man.css?5'; + title_ $o{title}.' - manned.org'; + }; + body_ sub { + header_ sub { + a_ href => '/', 'Manned.org'; + form_ action => '/browse/search', method => 'get', sub { + input_ type => 'text', name => 'q', id => 'q', tabindex => 1; + input_ type => 'submit', value => 'Search'; + } + }; + main_ class => $o{mainclass}, $content; + footer_ sub { + span_ sub { + a_ href => '/info/about', 'about'; txt_ ' | '; + a_ href => 'mailto:manned@yorhel.nl', 'contact'; txt_ ' | '; + a_ href => 'https://code.blicky.net/yorhel/manned', 'source'; + }; + span_ 'all manual pages are copyrighted by their respective authors.'; + }; + script_ type => 'text/javascript', src => '/man.js', ''; + } + }; + + # write the SQL queries as a HTML comment when debugging is enabled + # (stolen from VNDB code) + # (TODO: Move this into TUWF or something) + if(tuwf->debug) { + my(@sql_r, @sql_i) = (); + for (tuwf->{_TUWF}{DB}{queries}->@*) { + my($sql, $params, $time) = @$_; + my @params = sort { $a =~ /^[0-9]+$/ && $b =~ /^[0-9]+$/ ? $a <=> $b : $a cmp $b } keys %$params; + my $prefix = sprintf " [%6.2fms] ", $time*1000; + push @sql_r, sprintf "%s%s | %s", $prefix, $sql, join ', ', map "$_:".DBI::neat($params->{$_}), @params; + my $i=1; + push @sql_i, $prefix.($sql =~ s/\?/tuwf->dbh->quote($params->{$i++})/egr); + } + my $sql_r = join "\n", @sql_r; + my $sql_i = join "\n", @sql_i; + my $modules = join "\n", sort keys %INC; + lit_ "\n<!--\nSQL (with placeholders):\n$sql_r\n\nSQL (interpolated, possibly buggy):\n$sql_i\n\nMODULES:\n$modules\n-->"; + } +} - my $mans = $self->dbManInfo(package => $sel->{id}, results => 200, page => $f->{p}, sort => 'syspkgname'); - my $count = $self->dbManInfo(package => $sel->{id}, countonly => 1)->[0]{count}; - # Latest version of this package determines last modification date of the page. - $self->setLastMod($vers->[0]{released}); +sub paginate_ { + my($url, $count, $perpage, $p) = @_; + return if $count <= $perpage; - my $title = "$sys->{name}".($sys->{release}?" $sys->{release}":"")." / $pkg->{category} / $pkg->{name}"; - $self->htmlHeader(title => "$title $sel->{version}"); - h1 $title; + my sub l_ { + my($c)= @_; + a_ href => "$url$c", $c if $c != $p; + b_ $c if $c == $p; + }; - div id => 'pkgversions'; - h2 'Versions'; - ul; - for(@$vers) { - li; - a href => "/pkg/$sys->{short}/$pkg->{category}/$pkg->{name}/$_->{version}", $_->{version} if $_ != $sel; - b " $_->{version}" if $_ == $sel; - i " $_->{released}"; - end; - } - end; - end; - - div id => 'pkgmans'; - h2 "Manuals for version $sel->{version}"; - paginate "/pkg/$sys->{short}/$pkg->{category}/$pkg->{name}/$sel->{version}?p=", $count, 200, $f->{p}; - ul; - for(@$mans) { - li; - a href => "/$_->{name}/".substr($_->{hash},0,8), "$_->{name}($_->{section})"; - b " $_->{locale}" if $_->{locale}; - i " $_->{filename}"; - end; + my $lp = ceil($count/$perpage); + nav_ class => 'paginate', sub { + l_ 1 if $p > 1+4; + b_ '...' if $p > 1+5; + l_ $_ for (($p > 4 ? $p-4 : 1)..($p+4 > $lp ? $lp : $p+4)); + b_ '...' if $p < $lp-5; + l_ $lp if $p < $lp-4; } - end; - paginate "/pkg/$sys->{short}/$pkg->{category}/$pkg->{name}/$sel->{version}?p=", $count, 200, $f->{p}; - end; - - $self->htmlFooter; } -sub man_redir { - my($self, $sys, $path) = @_; +TUWF::set error_404_handler => sub { + tuwf->resStatus(404); + my $title = 'No manual entry for '.tuwf->reqPath; + framework_ title => $title, sub { + h1_ $title; + p_ 'That is, the page you were looking for doesn\'t exist.'; + }; +}; - # Path can be: - # 1. <name> - # 2. <category>/<package>/<name> - # 3. <category>/<package>/<version>/<name> - $sys = $self->{sysbyshort}{$sys}; - return $self->resNotFound if !$sys; +TUWF::get '/' => sub { + my $stats = tuwf->dbRow('SELECT * FROM stats_cache'); + + sub num { local $_=shift; 1 while(s/(\d)(\d{3})($|,)/$1,$2/); $_ }; + + framework_ title => 'Man Pages Archive', mainclass => 'thin', sub { + h1_ 'Welcome to Manned.org'; + h2_ 'The archive for man pages'; + lit sprintf <<' _', map num($stats->{$_}), qw|hashes mans files packages|; + <p> + Indexing <b>%s</b> versions of <b>%s</b> manual pages found in + <b>%s</b> files of <b>%s</b> packages. + </p><p> + Manned.org aims to index all manual pages from a variety of systems, both + old and new, and provides a convenient interface for looking up and viewing + the various versions of each man page. + <a href="/info/about">More information »</a> + </p> + _ + + h2_ 'Indexed systems'; + div_ class => 'systems', sub { + my %sys; + push $sys{$_->{name}}->@*, $_ for systems->@*; + div_ sub { + my $sys = $sys{$_}; + my $img = $sys->[0]{short} =~ s/^(.+)-.+$/$1/r; + if(@$sys == 1) { + a_ href => "/pkg/$sys->[0]{short}", sub { + img_ width => 50, height => 50, src => "images/$img.png"; + b_ $sys->[0]{name}; + }; + return; + } + img_ width => 50, height => 50, src => "images/$img.png"; + div_ sub { + b_ $sys->[0]{name}; + for(reverse @$sys) { + a_ href => "/pkg/$_->{short}", $_->{release}; + lit_ ' '; + } + }; + } for sort keys %sys; + }; + + h2_ 'Other relevant sites'; + ul_ sub { + li_ sub { a_ href => 'http://man7.org/linux/man-pages/index.html', 'man7.org'; txt_ ' - Linux man pages from several upstream projects.' }; + li_ sub { a_ href => 'https://manpag.es/', 'ManPag.es'; txt_ ' - Man pages from several Linux distributions.' }; + li_ sub { a_ href => 'https://www.mankier.com/', 'ManKier'; txt_ ' - Fedora Rawhide + some manually imported man pages; Nicely formatted and with some unique features.' }; + li_ sub { a_ href => 'https://man.cx/', 'man.cx'; txt_ ' - Man pages extracted from Debian testing.' }; + li_ sub { a_ href => 'http://man.he.net/', 'man.he.net'; txt_ ' - Also seems to be from a Debian-like system.' }; + li_ sub { a_ href => 'https://linux.die.net/man/', 'die.net'; txt_ ' - Seems to be based on an RPM-based Linux distribution.' }; + li_ sub { a_ href => 'http://manpages.org/', 'manpages.org'; txt_ ' - Lots of mostly-nicely formatted man pages, no clue about source.' }; + li_ sub { a_ href => 'https://www.manpagez.com/', 'manpagez.com'; txt_ ' - Mac OS X, has some GTK-html and texinfo documentation as well.' }; + li_ sub { a_ href => 'https://man.archlinux.org/', 'Arch Linux Man Pages' }; + li_ sub { a_ href => 'https://manpages.debian.org/', 'Debian Man Pages' }; + li_ sub { a_ href => 'https://www.dragonflybsd.org/cgi/web-man', 'DragonFlyBSD Man Pages' }; + li_ sub { a_ href => 'https://www.freebsd.org/cgi/man.cgi', 'FreeBSD.org Man Pages' }; + li_ sub { a_ href => 'https://man.netbsd.org/', 'NetBSD Man Pages' }; + li_ sub { a_ href => 'https://www.openbsd.org/cgi-bin/man.cgi', 'OpenBSD Man Pages' }; + li_ sub { a_ href => 'https://manpages.ubuntu.com/', 'Ubuntu Manuals' }; + li_ sub { a_ href => 'https://man.voidlinux.org/', 'Void Linux manpages' }; + }; + }; +}; - my $man; - if($path !~ m{/}) { # (1) - ($man) = $self->dbManPrefName($path, sysid => $sys->{id}); - } else { - $path =~ s{/([^/]+)$}{}; - my $name = $1; +TUWF::get '/info/about' => sub { + framework_ title => 'About', mainclass => 'thin', sub { + h1_ 'About Manned.org'; + lit <<' _'; + <h2 id="goal">Goal</h2> + <p> + The state of online indices of manual pages used to be a sad one. Existing + sites used to only offer you a single version of a man page: From one + origin, and often only in a single language. Most didn't even tell you + where the manual actually originated from, making it very hard to + determine whether the manual you found applied to your situation and even + harder to find a manual for a specific system. Additionally, some sites + rendered the manuals in an unreadable way, didn't correctly handle special + formatting - like tables - or didn't correctly display non-ASCII + characters. + </p><p> + Nowadays there are many good alternatives, but Manned.org was one of the + sites created in order to improve that situation. This site aims to index + the manual pages from a variaty of systems, both old and new, and allows you + to browse through the various versions of a manual page to find out how each + system behaves. The manuals are stored in the database as UTF-8, and are + passed through <a href="http://www.gnu.org/software/groff/">groff</a> to + render them in (mostly) the same way as they are displayed in your terminal. + </p><p> + This website is <a href="https://code.blicky.net/yorhel/manned">open + source</a> (AGPL licensed) and written in a combination of Perl and Rust. + The entire PostgreSQL database is available + <a href="#database-download">for download</a>. + </p> + + <h2 id="url-format">URL format</h2> + <p>You can link to specific packages and man pages with several URL formats. + These URLs will keep working in the future, so you should not have to worry + about eventual dead links.</p> + <h3>Man pages</h3> + <p>The following URLs are available to refer to an individual man page:</p> + <dl> + <dt><code>/<name>[.<section>]</code> or <code>/man/<name>[.<section>]</code></dt><dd> + Will try to get the latest and most-close-to-upstream version of a man + page. Note that this will fetch the man page from any of the available + systems, so may result in confusing scenarios for system-specific + documentation. I try to at least keep the selection algorithm stable and + deterministic, but can't provide any guarantees. Examples:<br> + <a href="/socket">/socket</a><br> + <a href="/socket.7">/socket.7</a><br> + <a href="/man/socket.7">/man/socket.7</a></dd> + <dt><code>/man/<system>/<name>[.<section>]</code></dt><dd> + Will get the latest version of a man page from the given system, e.g.:<br> + <a href="/man/ubuntu/rsync">/man/ubuntu/rsync</a><br> + <a href="/man/ubuntu-xenial/rsync">/man/ubuntu-xenial/rsync</a></dd> + <dt><code>/man/<system>/<category>/<package>/<name>[.<section>]</code></dt><dd> + Will get the latest version of a man page from the given package, e.g.:<br> + <a href="/man/ubuntu-xenial/net/rsync/rsync">/man/ubuntu-xenial/net/rsync/rsync</a></dd> + <dt><code>/man/<system>/<category>/<package>/<version>/<name>[.<section>]</code></dt><dd> + Will get the man page from a specific package version, e.g.:<br> + <a href="/man/ubuntu-xenial/net/rsync/3.1.1-3ubuntu1/rsync">/man/ubuntu-xenial/net/rsync/3.1.1-3ubuntu1/rsync</a></dd> + <dt><code>/man.<language>/...</code></dt><dd> + Adding a language code to the <code>/man/</code> component will select + the man page in the requested language. The man page has to be available + in that language, otherwise you will get a 404. Redirects to other + languages as fallback may be implemented in the future. English man + pages are typically not tagged with a language at all, so explicitely + requesting <code>/man.en/...</code> will usually fail. This, too, may be + improved in the future. Examples:<br> + <a href="/man.de/faked-tcp">/man.de/faked-tcp</a><br> + <a href="/man.fr/fedora/rsync.1">/man.de/fedora/rsync.1</a></dd> + <dt><code>/man.<8-hex-digits>/...</code></dt><dd> + Permalink format. Adding the shorthash of the man page to the + <code>/man/</code> component of the above URLs will get that specific + man page from the requested system and/or package. The contents of the + man page should generally be the same regardless of which system or + package is included in the URL, but the UI may provide a different + nagivation context. Examples:<br> + <a href="/man.910be0ed/ls">/man.910be0ed/ls</a><br> + <a href="/man.910be0ed/fedora/ls">/man.910be0ed/fedora/ls</a><br> + <a href="/man.910be0ed/arch/ls">/man.910be0ed/arch/ls</a><br> + <a href="/man.910be0ed/fedora/everything/coreutils-common/ls">/man.910be0ed/fedora/everything/coreutils-common/ls</a></dd> + <dt><code>/raw...</code></dt><dd> + In all of the above URL formats, you can change <code>/man</code> with + <code>/raw</code> to get the raw UTF-8 encoded man page source, e.g.:<br> + <a href="/raw/socket.7">/raw/socket.7</a><br> + <a href="/raw/ubuntu-xenial/net/rsync/3.1.1-3ubuntu1/rsync">/raw/ubuntu-xenial/net/rsync/3.1.1-3ubuntu1/rsync</a><br> + <a href="/raw.de/faked-tcp">/raw.de/faked-tcp</a><br> + <a href="/raw.910be0ed/fedora/ls">/raw.910be0ed/fedora/ls</a></dd> + <dt><code>/<name>/<8-hex-digits></code></dt><dd> + Old permalink format for a specific man page (e.g. <a href="/ls/910be0ed">/ls/910be0ed</a>).</dd> + </dl> + <p>In all URLs where an optional <code>.<section></code> can be provided, + the search is performed as a prefix match. For example, <a + href="/cat.3">/cat.3</a> will provide the <code>cat.3tcl</code> man page if + no exact <code>cat.3</code> version is available. Linking to the full + section name is also possible: <a href="/cat.3tcl">/cat.3tcl</a>. If no + section is given and multiple sections are available, the lowest section + number is chosen.</p> + <h3>Packages</h3> + <p>Linking to individual packages is also possible. These pages will show a + listing of all manual pages available in the given package.</p> + <dl> + <dt><code>/pkg/<system>/<category>/<package></code></dt><dd> + For the latest version of a package (e.g. <a + href="/pkg/arch/core/coreutils">/pkg/arch/core/coreutils</a>).</dd> + <dt><code>/pkg/<system>/<category>/<package>/<version></code></dt><dd> + For a particular version of a package (e.g. <a + href="/pkg/arch/core/coreutils/8.25-2">/pkg/arch/core/coreutils/8.25-2</a>).</dd> + </dl> + <p>This site only indexes packages that actually have manual pages, + linking to a package that doesn't have any will result in a 404 page.</p> + + <h2 id="indexing">The indexing process</h2> + <p> + All man pages are fetched right from the (binary) packages available on the + public repositories of Linux distributions. In particular:<br /> + </p> + <dl> + <dt>Alpine Linux</dt><dd> + The main (since 3.0) and community (since 3.3) repositories are indexed + for the x86_64 architecture. Indexing started in December 2021, packages + and releases not available in the repositories at that time have not + been indexed. I haven't found an archive for version 2.x releases + yet.</dd> + <dt>Arch Linux</dt><dd> + The core, extra and community repositories are fetched from a local + Arch mirror. Indexing started around begin June 2012. The i686 + architecture was indexed until November 6th, 2016, packages after that + were fetched from from x86_64.</dd> + <dt>Debian</dt><dd> + Historical releases were fetched from <a + href="http://archive.debian.org/debian/">http://archive.debian.org/debian/</a> + and <a href="http://snapshot.debian.org/">http://snapshot.debian.org/</a>. + For buzz, rex and bo, we're missing a few man pages because some packages + were missing from the repository archives. Where available, all components + (main, contrib and non-free) from the $release and $release-updates + repositories are indexed.</dd> + <dt>CentOS</dt><dd> + Historical releases were fetched from <a + href="http://vault.centos.org/">vault.centos.org</a>, current releases + from a local mirror. Where applicable, the following repositories were + indexed: addons, centosplus, contrib, extras, os. The i386 architecture + was indexed for versions lower than 7.0, since 7.0 the packages from + x86_64 are indexed. + <dt>Fedora</dt><dd> + Historical releases were fetched from <a + href="http://archives.fedoraproject.org/pub/archive/fedora/linux/">archives.fedoraproject.org</a>, + current releases from a local repository. Fedora Core 1 till 6 are + (incorrectly) called 'Fedora' here. To compensate for that, Fedora 3 till + 6 also include the Extras repository. For Fedora 7 and later, the + 'Everything' and 'updates' repositories are indexed. The i386 arch was + indexed for Fedora 17 and older, the x86_64 arch starting with Fedora + 18.</dd> + <dt>FreeBSD</dt><dd> + Historical releases were fetched from <a + href="http://ftp-archive.freebsd.org/mirror/FreeBSD-Archive/">http://ftp-archive.freebsd.org/mirror/FreeBSD-Archive/</a>. + The base installation tarballs are included in the database as packages + prefixed with <i>core-</i>. The package repositories have also been + indexed, except for 2.0.5 - 2.2.7 and 3.0 - 3.3 because those were not + available on the ftp archive. Only the -RELEASE repositories have been + included, which is generally a snapshot of the ports directory around the + time of the release. The release dates indicated for many packages were + guessed from the file modification dates in the tarball, and may be + inaccurate. The i368 arch was indexed for FreeBSD 11.0 and older, the + amd64 arch starting with 11.1.</dd> + <dt>NetBSD</dt><dd> + Only the core installation sets have been indexed, <a + href="https://www.pkgsrc.org/">pkgsrc</a> is awesome but out of scope + for now. The i368 arch was indexed for 5.x and older, the amd64 arch + starting with 6.0. Releases before 1.3 only distributed preformatted man + pages and have therefore not been indexed. The original roff sources + could perhaps be extracted from the source tarballs, but that's a + project for another time.</dd> + <dt>Ubuntu</dt><dd> + Historical releases were fetched from <a + href="http://old-releases.ubuntu.com/ubuntu/">http://old-releases.ubuntu.com/ubuntu/</a>, + supported releases from a local mirror. All components (main, universe, + restricted and multiverse) from the $release, $release-updates and + $release-security repositories are indexed. Indexing started around mid + June 2012. All releases before 2017 were indexed from the i386 + repositories, starting with 17.04 the amd64 repositories were used.</dd> + </dl> + <p> + Only packages for a single architecture (i386 or amd64) are scanned. To my + knowledge, packages that come with different manuals for different + architectures either don't exist or are extremely rare. It does happen that + some packages are not available for all architectures. Usually, though, + every package is at least available for the most popular architecture, so + hopefully we're not missing out on much. <br /><br /> + The repositories are scanned for new packages on a daily basis. + </p> + + <h2 id="database-download">Database download</h2> + <p> + This site is backed by a PostgreSQL database containing all the man pages. + Weekly dumps of the full database are available for download at + <a href="http://dl.manned.org/dumps/">http://dl.manned.org/dumps/</a>. + <br /><br /> + Be warned that the download server may not be terribly fast or reliable, + so it is advisable to use a client that supports resumption of partial + downloads. See <a href="/wget">wget's -c</a> or + <a href="/curl">curl's -C</a>. + <br /><br /> + The database schema is "documented" at <a + href="https://code.blicky.net/yorhel/manned/src/branch/master/sql/schema.sql">schema.sql</a> + in the git repo. Keep in mind that these dumps don't constitute a stable + API and, while this won't happen frequently, incompatible schema changes + or Postgres major version bumps will occassionally occur. + </p> + + <h2 id="future-plans">Future plans</h2> + <p> + This site isn't nearly as awesome yet as it could be. Here's some ideas that + would be nice to have in the future: + <ul> + <li>Index a few more systems: Gentoo (now that it has official binary packages), OpenBSD and perhaps others.</li> + <li>Better browsing and discovery features.</li> + <li>Improved, more intelligent, search,</li> + <li><a href="/apropos.1">apropos(1)</a> emulation(?),</li> + <li>Diffs between various versions of a man page,</li> + <li>Anchor links within man pages, for easier linking to a section or paragraph,</li> + <li>Alternative formats (Text, PDF, more semantic HTML, etc),</li> + <li>A command-line client, like <a href="/man.1">man(1)</a> with manned.org as database backend.</li> + </ul> + </p> + + <h2 id="copyright">Copyright</h2> + <p> + All manual pages are copyrighted by their respective authors. The manuals + have been fetched from publically available repositories of free and + (primarily) open source software. The distributors of said software have put + in efforts to only include software and documentation that allows free + distribution. Nonetheless, if a manual that does not allow to be + redistributed has been inadvertently included in our index, please let me + know and I will have it removed as soon as possible. + </p> + _ + }; +}; - my($pkg, $ver) = pkg_frompath($self, $sys->{id}, $path); # Handles (2) and (3) - return $self->resNotFound if !$pkg; - my $verid = $ver && $self->dbPackageVersions($pkg->{id}, $ver)->[0]{id}; - return $self->resNotFound if $ver && !$verid; +# Very simple (and fast) prefix match. +sub search_man { + my($q, $limit) = @_; + + my $sect = $q =~ s/^([0-9])\s+// || $q =~ s/\(([a-zA-Z0-9]+)\)$// ? $1 : ''; + my $name = $q =~ s/^([a-zA-Z0-9,.:_-]+)// ? $1 : ''; + + return !$name ? [] : tuwf->dbAlli( + 'SELECT name, section FROM mans WHERE', sql_and( + sql('lower(name) LIKE', \(escape_like(lc $name).'%')), + $sect ? sql('section ILIKE', \(escape_like(lc $sect).'%')) : (), + ), 'ORDER BY name, section LIMIT', \$limit, + ); +} - ($man) = $self->dbManPrefName($name, sysid => $sys->{id}, pkgid => $pkg->{id}, pkgver => $verid); - } - return $self->resNotFound if !$man; - $self->resRedirect("/$man->{name}/".substr($man->{hash}, 0, 8), 'temp'); +TUWF::get '/browse/search' => sub { + my $q = tuwf->reqGet('q')||''; + my $man = search_man $q, 150; + return tuwf->resRedirect("/$man->[0]{name}.$man->[0]{section}", 'temp') if @$man == 1; + + framework_ title => 'Search results for '.$q, mainclass => 'searchres', sub { + h1_ 'Search results for '.$q; + # Package search would also be useful. + p_ 'Note: This is just a simple case-insensitive prefix match on the man names. In the future we\'ll have more powerful search functionality. Hopefully.'; + if(@$man) { + ul_ sub { + li_ sub { + a_ href => "/$_->{name}.$_->{section}", $_->{name}; + small_ " $_->{section}"; + } for @$man; + } + } else { + p_ 'No results :-('; + } + }; }; -sub _man_nav { - my($self, $man, $toc) = @_; +TUWF::get '/xml/search.xml' => sub { + my $q = tuwf->reqGet('q')||''; + my $man = search_man $q, 20; + + tuwf->resHeader('Content-Type' => 'text/xml; charset=UTF-8'); + xml; + tag 'results', sub { + tag 'item', id => "$_->{name}.$_->{section}", %$_, undef for @$man; + }; +}; - my @sect = $self->dbManSections($man->{name}); - my @lang = $self->dbManLanguages($man->{name}, $man->{section}); - return if !@sect && !@lang && !@$toc; - # TODO: This is ugly, especially because clicking on a translation or - # section, you can end up with a man page that is nowhere close to the man - # page you're currently reading. Opening a version selector box might be a - # better alternative. +# Object to represent the various URLs to a man page. +# +# Parameters: +# fmt => man|txt|raw +# shorthash => 8-char hex +# lang => language code +# system => system shortname +# category => package category +# package => name of the package +# version => package version +# man => name of the man page +# section => man page section +# +# URL format: +# /$fmt[.$shorthash][.$lang][/$system[/$category/$package[/$version]]]/$man[.$section] +# +# Note that the URL format has some ambiguity: +# - $category may contain a slash, so a database lookup is required to +# disambiguate between URLs with [/$version] and those without. +# - $man may contain a dot, so a database lookup is required to disambiguate +# between URLs with [.$section] and those without +# +# $system may also refer to system shortnames without the version suffix (e.g. +# 'ubuntu' rather than 'ubuntu-impish'). In that case the man page from the +# latest release of that system is chosen. +package ManUrl { + sub new { my($p,%o)=@_; bless \%o, $p } + sub set { my($o,@o)=@_; bless +{%$o,@o}, ref $o } + sub mansect { $_[0]{man}.(defined $_[0]{section} ? ".$_[0]{section}" : '') } + use overload '""' => sub { + my($o)=@_; + "/$o->{fmt}".(defined $o->{shorthash} ? ".$o->{shorthash}" : '').(defined $o->{lang} ? ".$o->{lang}" : '') + .(defined $o->{system} ? ("/$o->{system}" + .(defined $o->{category} ? ("/$o->{category}/$o->{package}" + .(defined $o->{version} ? "/$o->{version}" : '')) : '')) : '') + .'/'.$o->mansect + }; +}; - div id => 'nav'; - if(@sect > 1) { - b 'Sections'; - p; - for (@sect) { - if($man->{section} eq $_) { - i $_; - } else { - a href => "/$man->{name}.$_", $_; - } - txt ' '; - } - end; - } - - if(@lang > 1) { - b 'Languages'; - p; - (my $cur = $man->{locale}||'') =~ s/\..*//; - for (@lang) { - if(($_||'') eq $cur) { - i $_ || 'default'; - } else { - a href => $_ ? "/lang/$_/$man->{name}.$man->{section}" : "/$man->{name}.$man->{section}", $_ || 'default'; +sub man_nav_ { + my($man, $url, $toc, $htmllang) = @_; + + my @systems = tuwf->dbAlli(' + SELECT DISTINCT p.system + FROM packages p + JOIN package_versions v ON v.package = p.id + JOIN files f ON f.pkgver = v.id + JOIN mans m ON m.id = f.man + WHERE m.name =', \$man->{name}, 'AND m.section =', \$man->{section} + )->@*; + + my @sect = map $_->{section}, tuwf->dbAlli( + 'SELECT DISTINCT section FROM mans WHERE name =', \$man->{name}, 'ORDER BY section' + )->@*; + + my @lang = map $_->{lang}, tuwf->dbAlli( + "SELECT DISTINCT substring(l.locale from '^[^.]+') AS lang + FROM files f + JOIN mans m ON m.id = f.man + JOIN locales l ON l.id = f.locale + WHERE m.name =", \$man->{name}, 'AND m.section =', \$man->{section}, " + ORDER BY substring(l.locale from '^[^.]+') NULLS FIRST" + )->@*; + + nav_ sub { + form_ action => '/sysredir/'.$url->mansect(), method => 'get', + onsubmit => 'location.href="/man/"+system_select[system_select.selectedIndex].value+"/'.$url->mansect().'";return false', + sub { + my %names; + push $names{$_->{name}}->@*, $_ for map sysbyid->{$_->{system}}, sort { $b->{system} <=> $a->{system} } @systems; + select_ id => 'system_select', name => 'system', sub { + for (sort { ($names{$b}->@* == 1) <=> ($names{$a}->@* == 1) || $a cmp $b } keys %names) { + my $s = $names{$_}; + if (@$s == 1) { + option_ value => $s->[0]{short}, selected => $s->[0]{id} == $man->{system}?'':undef, $s->[0]{full}; + next; + } + optgroup_ label => $_, sub { + option_ value => $_->{short}, selected => $_->{id} == $man->{system}?'':undef, $_->{full} for @$s; + }; + } + }; + input_ type => 'submit', value => 'Go'; + } if @systems > 1; + + # TODO: This is ugly, especially because clicking on a translation or + # section, you can end up with a man page that is nowhere close to the + # man page you're currently reading. Sections or languages available + # for the currently selected system should be highlighted. + if(@sect > 1) { + b_ 'Sections'; + p_ sub { + for (@sect) { + if($man->{section} eq $_) { + i_ $_; + } else { + a_ href => "/man/$man->{name}.$_", $_; + } + txt_ ' '; + } + } } - txt ' '; - } - end; - } - - if(@$toc > 1) { - b 'Table of Contents'; - ul; - for (0..$#$toc) { - li; - a href => sprintf('#head%d', $_+1), lc $toc->[$_]; - end; - } - end; - } - end; -} + if(@lang > 1) { + b_ 'Languages'; + p_ sub { + (my $cur = $man->{locale}||'') =~ s/\..*//; + for (@lang) { + if(($_||'') eq $cur) { + i_ $_ || 'default'; + } else { + a_ href => $_ ? "/man.$_/$man->{name}.$man->{section}" : "/man/$man->{name}.$man->{section}", $_ || 'default'; + } + txt_ ' '; + } + } + } -sub _normalizename { - local $_ = shift; - # Firefox seems to escape [ and ] in URLs. It doesn't really have to... - s/%5b/[/ig; - s/%5d/]/ig; - # Man pages with spaces in the path, eww - s/%20/ /g; - $_; + if(@$toc > 1) { + b_ 'Table of Contents'; + ul_ sub { + for (0..$#$toc) { + li_ sub { + a_ @$htmllang, href => sprintf('#head%d', $_+1), sub { lit_ lc $toc->[$_] }; + } + } + } + } + } } # Replace .so's in man source with the contents (if available in the same # package) or with a reference to the other man page. sub soelim { - my($self, $verid, $src) = @_; - - # tix comes with[1] a custom(?) macro package. But it looks okay even without - # loading that. - # [1] It actually doesn't, the tcllib package appears to have that file, but - # doesn't '.so' it. - $src =~ s/^\.so man.macros$//mg; - - # Other .so's should be handled by html() - $src =~ s{^\.so (.+)$}{ - my $path = $1; - my $name = (reverse split /\//, $path)[0]; - my($man) = $verid ? $self->dbManPrefName($name, pkgver => $verid) : (); - if($man) { - # Recursive soelim, but the second call gets $verid=0 so we don't keep checking the database - soelim($self, 0, $self->dbManContent($man->{hash})) - } else { - ".in -10\n.sp\n\[\[\[MANNEDINCLUDE$path\]\]\]" - } - }emg; - return $src; -} - - -sub man { - my($self, $name, $hash) = @_; - - $name = _normalizename($name); - - # Unfortunately, even in the permalink format with the hash, we don't know - # from which package we're supposed to get the man page. This info is - # needed in order to do .so substitution, so we can substitute files from - # the same package as the requested man page. Use the dbManPref logic here - # to deterministically select a good package. - my($man, undef) = $hash - ? $self->dbManPref(name => $name, shorthash => $hash) - : $self->dbManPrefName($name); - return $self->resNotFound() if !$man; - - my $fmt = ManUtils::html(ManUtils::fmt_block soelim $self, $man->{verid}, $self->dbManContent($man->{hash})); - my @toc; - $fmt =~ s{\n<b>(.+?)<\/b>\n}{ - push @toc, $1; - my $c = @toc; - qq{\n<a href="#head$c" id="head$c">$1</a>\n} - }eg; - - $self->setLastMod($man->{released}); - $self->htmlHeader(title => $name); - _man_nav($self, $man, \@toc); - div id => 'manbuttons'; - h1 $man->{name}; - ul 'data-hash' => $man->{hash}, 'data-name' => $man->{name}, 'data-section' => $man->{section}, 'data-locale' => $man->{locale}||'', - 'data-hasversions' => $self->dbManHasVersions($man->{name}, $man->{section}, $man->{locale}, $man->{hash}); - li; a href => "/$man->{name}/".substr($man->{hash}, 0, 8).'/src', 'source'; end; - li; a href => "/$man->{name}/".substr($man->{hash}, 0, 8), 'permalink'; end; - end; - end; - div id => 'manres', class => 'hidden'; - end; - - div id => 'contents'; - pre; lit $fmt; end; - end; - $self->htmlFooter(); -} - - -sub src { - my($self, $name, $hash) = @_; - - $name = _normalizename($name); - - my $m = $self->dbManInfo(name => $name, shorthash => $hash); - return $self->resNotFound if !@$m; - - $self->setLastMod($m->[0]{released}); - $self->resHeader('Content-Type', 'text/plain; charset=UTF-8'); - $self->resHeader('Content-Disposition', sprintf 'filename="%s.%s"', $m->[0]{name}, $m->[0]{section}); - my $c = $self->dbManContent($m->[0]{hash}); - lit $c; -} - - -sub xmlsearch { - my $self = shift; - my $q = $self->reqGet('q')||''; - my $man = $self->dbSearch($q, 20); - - # The JS dropdown search expects this particular format. - $self->resHeader('Content-Type' => 'text/xml; charset=UTF-8'); - xml; - tag 'results'; - tag 'item', id => "$_->{name}.$_->{section}", %$_, undef for(@$man); - end 'results'; + my($verid, $src) = @_; + + # tix comes with* a custom(?) macro package. But it looks okay even without loading that. + # (* It actually doesn't, the tcllib package appears to have that file, but doesn't '.so' it) + $src =~ s/^\.so man.macros$//mg; + + # Other .so's should be handled by html() + $src =~ s{^\.so (.+)$}{ + my $path = $1; + my $name = (reverse split /\//, $path)[0]; + my($man) = $verid ? man_pref_name $name, sql 'v.id =', \$verid : (); + $man->{name} + # Recursive soelim, but the second call gets $verid=0 so we don't keep checking the database + ? soelim(0, tuwf->dbVali("SELECT content FROM contents WHERE id =", \$man->{content})) + : ".in -10\n.sp\n\[\[\[MANNEDINCLUDE$path\]\]\]" + }emg; + $src; } -sub jsontree { - my $self = shift; - - my $f = $self->formValidate( - { get => 'name', required => 0, maxlength => 256 }, - { get => 'section', required => 0, maxlength => 32 }, - { get => 'locale', required => 0, default => '', maxlength => 32 }, - { get => 'cur', required => 0, default => '', regex => qr/^[a-fA-F0-9]{40}$/ }, - { get => 'hash', required => 0, default => '', regex => qr/^[a-fA-F0-9]{40}$/ }, - ); - return $self->resNotFound() if $f->{_err} || (!$f->{hash} && !($f->{section} && $f->{name})); - - my $l = $self->dbManInfo(sort => 'syspkgname', $f->{hash} - ? (hash => $f->{hash}) - : (name => $f->{name}, section => $f->{section}, locale => $f->{locale})); - - # Convert the list into a tree - my $tree = []; - my($sys, $sysver, $pkg, $pkgver); - for my $m (@$l) { - my $sysname = $self->{sysbyid}{$m->{system}}{name}; - if(!$sys || $sysname ne $sys->{name}) { - $sys = { name => $sysname, childs => [] }; - $sysver = undef; - push @$tree, $sys; - } +sub man_page { + my($man, $url) = @_; + tuwf->resLastMod($man->{released}); - my $sysversion = $self->{sysbyid}{$m->{system}}{release} || ''; - if(!$sysver || $sysversion ne $sysver->{name}) { - $sysver = { name => $sysversion, childs => [] }; - $pkg = undef; - push @{$sys->{childs}}, $sysver; + my $content = tuwf->dbRowi('SELECT encode(hash, \'hex\') AS hash, content FROM contents WHERE id =', \$man->{content}); + if($url->{fmt} eq 'raw') { + tuwf->resHeader('Content-Type', 'text/plain; charset=UTF-8'); + tuwf->resHeader('Content-Disposition', sprintf 'filename="%s.%s"', $man->{name}, $man->{section}); + lit $content->{content}; + return; } - if(!$pkg || $m->{package} ne $pkg->{name}) { - $pkg = { name => $m->{package}, i => $m->{category}, table => [] }; - $pkgver = undef; - push @{$sysver->{childs}}, $pkg; + my $fmt = ManUtils::html ManUtils::fmt_block soelim $man->{verid}, $content->{content}; + if($url->{fmt} eq 'txt') { + # TODO: The 'txt' format is kind of broken right now as it includes our HTML formatting codes. + # This feature is a WIP and not advertised at the moment, anyway. + tuwf->resHeader('Content-Type', 'text/plain; charset=UTF-8'); + tuwf->resHeader('Content-Disposition', sprintf 'filename="%s.%s.txt"', $man->{name}, $man->{section}); + lit $fmt; + return; } - push @{$pkg->{table}}, [ - $pkgver && $pkgver eq $m->{version} ? {name=>''} : - {name => $m->{version}, href => "/pkg/$self->{sysbyid}{$m->{system}}{short}/$m->{category}/$m->{package}/$m->{version}"}, - { name => "$m->{name}($m->{section})", - $f->{hash} || lc($m->{hash}) eq lc($f->{cur}) ? () - : (href => sprintf('/%s/%s', $m->{name}, substr $m->{hash}, 0, 8)) - }, - { name => substr($m->{hash}, 0, 8), - $f->{hash} || lc($m->{hash}) eq lc($f->{cur}) ? () - : (href => sprintf('/%s/%s', $m->{name}, substr $m->{hash}, 0, 8)) - }, - { name => $m->{filename} } - ]; - $pkgver = $m->{version}; - } - - # Determine which elements to show/hide by default. - # It might make more sense to do this in JS, but since I am utterly - # incapable of writing maintainable JS I'm doing it here in order to keep the - # JS stupid and simple. - # TODO: Highlight systems/packages where the 'current' man page is? - for my $sys (@$tree) { - $sys->{expand} = 1 if $sys->{childs}[0]{name}; # Expand all systems that have named versions - $sys->{expand} = 1 if $f->{hash}; # Expand everything on 'location' - - my $i = 0; - for my $sysver (@{$sys->{childs}}) { - $i++; - $sysver->{expand} = 1 if !$sysver->{name}; # Expand unnamed versions (since you can't click them) - $sysver->{expand} = 1 if $f->{hash}; # Expand everything on 'location' - $sysver->{hide} = 1 if $i > 3 && @{$sys->{childs}} > 5; # Show only the first 3 versions - - for my $pkg (@{$sysver->{childs}}) { - $pkg->{expand} = 1 if @{$sysver->{childs}} <= 3; # Expand everything if there's not too many things to expand - $pkg->{expand} = 1 if $f->{hash}; # Expand everything on 'location' - - # TODO: Show/Hide duplicate hashes? - } - } - } - - # Why JSON? Because TUWF::XML is pretty slow with many nodes - $self->resHeader('Content-Type' => 'application/json; charset=UTF-8'); - lit(JSON::XS->new->ascii->encode($tree)); + # Prefix links to other man pages with the current system, to ensure we + # grab the most relevant man page. + # XXX: This is a hack, prefixing is better done directly in ManUtils. + my $sys = sysbyid->{$man->{system}}{short}; + $fmt =~ s{<a href="/}{<a href="/man/$sys/}g; + + my @toc; + $fmt =~ s{\n<b>([^<\n]+?)<\/b>\n}{ + push @toc, $1; + my $c = @toc; + qq{\n<a href="#head$c" id="head$c">$1</a>\n} + }eg; + + my $hasversions = tuwf->dbVali( + 'SELECT 1 + FROM files f + JOIN mans m ON m.id = f.man + JOIN locales l ON l.id = f.locale + WHERE m.name =', \$man->{name}, 'AND m.section =', \$man->{section}, ' + AND l.locale =', \$man->{locale}, ' + AND f.shorthash <> ', \$man->{shorthash}, ' + LIMIT 1' + ); + my @htmllang = $man->{locale} =~ /^([a-z]{2,3})(?:_([A-Z]{2}))?(?:$|@|\.)/ ? (lang => $1.($2?"-$2":'')) : (); + + framework_ title => $man->{name}, mainclass => 'manpage', sub { + man_nav_ $man, $url, \@toc, \@htmllang; + # TODO: Replace the 'versions' and 'locations' functionality with non-JS alternatives. + div_ id => 'manbuttons', sub { + h1_ $man->{name}; + ul_ 'data-hash' => $content->{hash}, + 'data-name' => $man->{name}, + 'data-section' => $man->{section}, + 'data-locale' => $man->{locale}||'', + 'data-hasversions' => $hasversions?1:0, + sub { + li_ sub { a_ href => $url->set(fmt => 'raw'), 'source' }; + li_ sub { a_ href => $url->set(system => sysbyid->{$man->{system}}{short}, category => undef, shorthash => shorthash_to_hex $man->{shorthash}), 'permalink' }; + } + }; + div_ id => 'manres', class => 'hidden', ''; + pre_ @htmllang, sub { lit_ $fmt }; + }; } +# /<name>[.section] - short and handy catch-all URL for man pages +# /<name>/<shorthash> - old permalink format +# This one has to go before the other mappings, to ensure that links work for +# man pages called 'pkg' or 'man'. +TUWF::get qr{/(?<name>[^/]+)(?:/(?<hash>[0-9a-f]{8}))?} => sub { + my $name = normalize_name tuwf->capture('name'); + my $shorthash = tuwf->capture('hash'); -package TUWF::Object; - -use TUWF ':html', 'html_escape'; -use Time::Local 'timegm'; + my($man, $sect) = man_pref_name $name, $shorthash ? sql 'f.shorthash =', \shorthash_to_int $shorthash : 'true'; + return tuwf->resNotFound() if !$man->{name}; -sub escape_like { - (my $v = shift) =~ s/([_%])/\\$1/g; - $v; -} + man_page $man, ManUrl->new( + fmt => 'man', + man => length $sect ? $man->{name} : $name, + section => length $sect ? $sect : undef, + ); +}; -sub htmlHeader { - my $self = shift; - my %o = @_; +# /<name>/<shorthash>/src - old URL format to get the raw man page +TUWF::get qr{/([^/]+)/([0-9a-f]{8})/src} => sub { + my $name = normalize_name tuwf->capture(1); + my $shorthash = tuwf->capture(2); - html; - head; - Link rel => 'stylesheet', type => 'text/css', href => '/man.css?4'; - title $o{title}.' - manned.org'; - end 'head'; - body; + my($man) = man_pref_name $name, sql 'f.shorthash =', \shorthash_to_int $shorthash; + return tuwf->resNotFound if !$man->{name}; + man_page $man, ManUrl->new(fmt => 'raw', man => $name); +}; - div id => 'header'; - a href => '/', 'manned.org'; - form action => '/browse/search', method => 'get'; - input type => 'text', name => 'q', id => 'q', tabindex => 1; - input type => 'submit', value => ' '; - end; - end; - div id => 'body'; -} +TUWF::get qr{/(?<fmt>man|txt|raw)(?:\.(?<shorthash>[a-fA-F0-9]{8}))?(?:\.(?<lang>[^/]+))?/(?<path>.+)} => sub { + my($fmt, $shorthash, $lang, $path) = tuwf->captures(qw|fmt shorthash lang path|); + my @where; + my $name = normalize_name($path =~ s{/?([^/]+)$}{} && $1); + my $system = $path =~ s{^([^/]+)/?}{} && $1; -sub htmlFooter { - my($self, %o) = @_; - - br style => 'clear: both'; - end; - div id => 'footer'; - lit 'All manual pages are copyrighted by their respective authors. - | <a href="/info/about">About manned.org</a> - | <a href="mailto:contact@manned.org">Contact</a> - | <a href="https://code.blicky.net/yorhel/manned">Source</a>'; - end; - script type => 'text/javascript', src => '/man.js', ''; - end; - end 'html'; - - # write the SQL queries as a HTML comment when debugging is enabled - # (stolen from VNDB code) - if($self->debug) { - lit "\n<!--\n SQL Queries:\n"; - for (@{$self->{_TUWF}{DB}{queries}}) { - my $q = !ref $_->[0] ? $_->[0] : - $_->[0][0].(exists $_->[0][1] ? ' | "'.join('", "', map defined()?$_:'NULL', @{$_->[0]}[1..$#{$_->[0]}]).'"' : ''); - $q =~ s/^\s//g; - lit sprintf " [%6.2fms] %s\n", $_->[1]*1000, $q; + # $sys can be either a full system 'short' name, or a prefix (e.g. 'debian' meaning 'any debian-* version') + if($system) { + my $sysid = sysbyshort->{$system}; + $sysid = $sysid ? [$sysid->{id}] : [ map sysbyshort->{$_}{id}, grep /^\Q$system\E-/, keys sysbyshort->%* ]; + return tuwf->resNotFound if !@$sysid; + push @where, sql 'system IN', $sysid; } - lit "-->\n"; - } -} - - -# Set the last modification time from a string in yyyy-mm-dd format. -sub setLastMod { - my($s, $d) = @_; - return if $d !~ /^(\d{4})-(\d{2})-(\d{2})/; - my @t = gmtime timegm 0,0,0,$3,$2-1,$1; - $s->resHeader('Last-Modified', sprintf '%s, %02d %s %04d %02d:%02d:%02d GMT', - (qw|Sun Mon Tue Wed Thu Fri Sat|)[$t[6]], $t[3], - (qw|Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec|)[$t[4]], - $t[5]+1900, $t[2], $t[1], $t[0]); -} - - -sub dbManContent { - my($s, $hash) = @_; - return $s->dbRow(q{SELECT content FROM contents WHERE hash = decode(?, 'hex')}, $hash)->{content}; -} - - -# Options: name, section, shorthash, package, results, sort, countonly -sub dbManInfo { - my $s = shift; - my %o = ( - sort => '', - page => 1, - results => 10_000, - @_ - ); - - my %where = ( - $o{name} ? ('m.name = ?' => $o{name}) : (), - $o{package} ? ('m.package = ?' => $o{package}) : (), - defined($o{section}) ? ('m.section = ?' => $o{section}) : (), - $o{locale} ? ('m.locale = ?' => $o{locale}) : (), - defined($o{locale}) && !$o{locale} ? ('m.locale IS NULL' => 1) : (), - $o{shorthash} ? (q{substring(m.hash from 1 for 4) = decode(?, 'hex')} => $o{shorthash}) : (), - $o{hash} ? (q{m.hash = decode(?, 'hex')} => $o{hash}) : (), - ); - - my $order = - $o{sort} eq 'syspkgname' ? 'ORDER BY s.name, s.relorder DESC, p.name, v.released DESC, m.name, m.locale NULLS FIRST, m.filename' : ''; - - my $select = $o{countonly} ? 'COUNT(*) as count' - : "p.system, p.category, p.name AS package, v.version, v.released, v.id AS verid, m.name, m.section, m.filename, m.locale, encode(m.hash, 'hex') AS hash"; - - my($r, $np) = $s->dbPage(\%o, q{ - SELECT !s - FROM man m - JOIN package_versions v ON v.id = m.package - JOIN packages p ON p.id = v.package - JOIN systems s ON s.id = p.system - !W - !s - }, $select, \%where, $order); - wantarray ? ($r, $np) : $r; -} - -# Very simple (and fast) prefix match. -sub dbSearch { - my($s, $q, $limit) = @_; - - my $sect = $q =~ s/^([0-9])\s+// || $q =~ s/\(([a-zA-Z0-9]+)\)$// ? $1 : ''; - my $name = $q =~ s/^([a-zA-Z0-9,.:_-]+)// ? $1 : ''; - - return !$name ? [] : $s->dbAll( - 'SELECT name, section FROM man_index !W ORDER BY name, section LIMIT ?', - { - 'lower(name) LIKE ?' => escape_like(lc $name).'%', - $sect ? ('section ILIKE ?' => escape_like(lc $sect).'%') : (), - }, - $limit - ); -} - - -# Get the preferred man page for the given filters. Returns a row with the same fields as dbManInfo(). -sub dbManPref { - my($s, %o) = @_; - my %where = ( - length $o{name} ? ('m.name = ?' => $o{name}) : (), - $o{shorthash} ? (q{substring(m.hash from 1 for 4) = decode(?, 'hex')} => $o{shorthash}) : (), - length $o{section} ? ('m.section LIKE ?' => escape_like($o{section}).'%') : (), - $o{sysid} ? ('p.system = ?' => $o{sysid}) : (), - $o{package} ? ('p.id = ?' => $o{package}) : (), - $o{pkgver} ? ('v.id = ?' => $o{pkgver}) : (), - $o{language} ? (q{substring(locale from '^[^.]+') = ?} => $o{language}) : (), - ); - - # Criteria to determine a "preferred" man page: - # 1. english: English versions of a man page have preference over other locales - # 2. pkgver: Newer versions of the same package have preference over older versions - # 3. stdloc: Prefer man pages in standard locations - # 4. secmatch: Prefer an exact section match - # 5. arch: Prefer Arch over other systems (because it tends to be the most up-to-date, and closest to upstreams) - # 6. sysrel: Prefer a later system release over an older release - # 7. secorder: Lower sections before higher sections (because man does it this way, for some reason) - # 8. pkgdate: Prefer more recent packages (cross-distro) - # 9. Fall back on hash comparison, to ensure the result is stable - - $s->dbAll(q{ - WITH unfiltered AS ( - SELECT s AS sys, p AS pkg, v AS ver, m AS man - FROM man m - JOIN package_versions v ON v.id = m.package - JOIN packages p ON p.id = v.package - JOIN systems s ON s.id = p.system - !W - ), f_english AS( - SELECT * FROM unfiltered WHERE NOT EXISTS(SELECT 1 FROM unfiltered WHERE is_english_locale((man).locale)) OR is_english_locale((man).locale) - ), f_pkgver AS( - SELECT * FROM f_english a WHERE NOT EXISTS(SELECT 1 FROM f_english b WHERE (a.ver).package = (b.ver).package AND (a.ver).released < (b.ver).released) - ), f_stdloc AS( - SELECT * FROM f_pkgver WHERE NOT EXISTS(SELECT 1 FROM f_pkgver WHERE is_standard_man_location((man).filename)) OR is_standard_man_location((man).filename) - ), f_secmatch AS( - SELECT * FROM f_stdloc WHERE NOT EXISTS(SELECT 1 FROM f_stdloc WHERE (man).section = ?) OR (man).section = ? - ), f_arch AS( - SELECT * FROM f_secmatch WHERE NOT EXISTS(SELECT 1 FROM f_secmatch WHERE (sys).id = 1) OR (sys).id = 1 - ), f_sysrel AS( - SELECT * FROM f_arch a WHERE NOT EXISTS(SELECT 1 FROM f_arch b WHERE (a.sys).name = (b.sys).name AND (a.sys).relorder < (b.sys).relorder) - ), f_secorder AS( - SELECT * FROM f_sysrel a WHERE NOT EXISTS(SELECT 1 FROM f_sysrel b WHERE (a.man).section > (b.man).section) - ), f_pkgdate AS( - SELECT * FROM f_secorder a WHERE NOT EXISTS(SELECT 1 FROM f_secorder b WHERE (a.ver).released < (b.ver).released) - ) - SELECT (pkg).system, (pkg).category, (pkg).name AS package, (ver).version, (ver).released, (ver).id AS verid, - (man).name, (man).section, (man).filename, (man).locale, encode((man).hash, 'hex') AS hash - FROM f_pkgdate ORDER BY (man).hash LIMIT 1 - }, \%where, $o{section}||'', $o{section}||'')->[0]; -} - - -# Given the name of a man page with optional section, find out the actual name -# and section prefix of the man page and the preferred version. -sub dbManPrefName { - my($s, $name, %o) = @_; - - my $man = $s->dbManPref(%o, name => $name); - return ($man, '') if $man; - - return (undef, '') if $name !~ s/\.([^.]+)$//; - my $section = $1; - $man = $s->dbManPref(%o, name => $name, section => $section); - return ($man, $section) if $man; - return (undef, ''); -} - - -# Returns 1 of there are alternative versions of the given man page. -sub dbManHasVersions { - my($s, $name, $section, $locale, $hash) = @_; - return $s->dbRow( - q{SELECT 1 AS ok FROM man WHERE name = ? AND section = ? AND locale IS NOT DISTINCT FROM ? AND hash <> decode(?, 'hex') LIMIT 1}, - $name, $section, $locale, $hash - )->{ok}||0; -} + # $path is now either: + # 1. $category/$package + # 2. $cagegory/$package/$version + my($pkg, $ver) = length $path ? pkg_frompath sql_and(@where), $path : (undef,undef); + return tuwf->resNotFound if length $path && !$pkg; + push @where, sql 'p.id =', \$pkg->{id} if $pkg; + push @where, sql 'v.version =', \$ver if length $ver; + + push @where, sql 'f.shorthash =', \shorthash_to_int $shorthash if $shorthash; + push @where, sql 'l.locale ilike', \(escape_like($lang).'%') if $lang; + + my($man, $section) = man_pref_name $name, sql_and @where; + return tuwf->resNotFound if !$man; + + my $url = ManUrl->new( + fmt => $fmt, + shorthash => $shorthash, + lang => $lang, + system => length $system ? $system : undef, + category => $pkg ? $pkg->{category} : undef, + package => $pkg ? $pkg->{name} : undef, + version => length $ver ? $ver : undef, + man => length $section ? $man->{name} : $name, + section => length $section ? $section : undef, + ); + man_page $man, $url; +}; -# Returns all available languages for a man page -sub dbManLanguages { - my($s, $name, $section) = @_; - return map $_->{lang}, @{$s->dbAll(q{SELECT DISTINCT substring(locale from '^[^.]+') AS lang - FROM man WHERE name = ? AND section = ? - ORDER BY substring(locale from '^[^.]+') NULLS FIRST - }, $name, $section)}; -} +TUWF::get qr{/pkg/([^/]+)} => sub { + my $short = tuwf->capture(1); + + my $sys = sysbyshort->{$short}; + return tuwf->resNotFound if !$sys; + + my $f = tuwf->validate(get => + c => { onerror => 'all', enum => [ '0', 'all', 'a'..'z' ] }, + p => { onerror => 1, uint => 1, range => [1,200] }, + )->data; + + my $where = sql 'NOT dead AND system =', \$sys->{id}, $f->{c} ne 'all' ? ('AND match_firstchar(name,', \$f->{c}, ')') : (); + my $count = tuwf->dbVali('SELECT count(*) FROM', $packages_with_man, 'p WHERE', $where); + my $pkg = tuwf->dbPagei({ results => 200, page => $f->{p} }, + 'SELECT id, system, name, category, dead FROM', $packages_with_man, 'p WHERE', $where, 'ORDER BY name, category' + ); + + framework_ title => $sys->{full}, mainclass => 'pkglist', sub { + div_ sub { + div_ sub { + h1_ $sys->{full}; + }; + nav_ class => 'charselect', sub { + for('all', 0, 'a'..'z') { + a_ href => "/pkg/$short?c=$_", $_?uc$_:'#' if $_ ne $f->{c}; + b_ $_?uc$_:'#' if $_ eq $f->{c}; + } + }; + }; + small_ '(Packages without man pages are not listed)'; + + paginate_ "/pkg/$short?c=$f->{c};p=", $count, 200, $f->{p}; + ul_ sub { + li_ sub { + a_ href => "/pkg/$short/$_->{category}/$_->{name}", $_->{name}; + small_ ' '.$_->{category}; + } for @$pkg; + }; + paginate_ "/pkg/$short?c=$f->{c};p=", $count, 200, $f->{p}; + }; +}; -# Returns all available languages for a man page -sub dbManSections { - my($s, $name) = @_; - return map $_->{section}, @{$s->dbAll(q{SELECT DISTINCT section FROM man WHERE name = ? ORDER BY section}, $name)}; -} - +# Package info: /pkg/$system/$category/$name (/$version); $category may contain a slash, too. +TUWF::get qr{/pkg/([^/]+)/(.+)} => sub { + my ($short, $path) = tuwf->captures(1,2); + + my $sys = sysbyshort->{$short}; + return tuwf->resNotFound if !$sys; + + my($pkg, $ver) = pkg_frompath(sql('system =', \$sys->{id}), $path); + return tuwf->resNotFound if !$pkg; + + my $vers = tuwf->dbAlli(' + SELECT id, version, released + FROM package_versions v + WHERE package =', \$pkg->{id}, ' + AND EXISTS(SELECT 1 FROM files f WHERE f.pkgver = v.id) + ORDER BY released DESC' + ); + my $sel = $ver ? (grep $_->{version} eq $ver, @$vers)[0] : $vers->[0]; + return tuwf->resNotFound if !$sel; + + my $p = tuwf->validate(get => p => { onerror => 1, uint => 1, range => [1,100] })->data; + + my $count = tuwf->dbVali('SELECT count(*) FROM files WHERE pkgver =', \$sel->{id}); + my $mans = tuwf->dbPagei({ results => 200, page => $p }, ' + WITH lst AS ( + SELECT f.man, m.name, m.section, f.shorthash, f.filename, l.locale + FROM files f + JOIN locales l ON l.id = f.locale + JOIN mans m ON m.id = f.man + WHERE f.pkgver =', \$sel->{id}, ' + ), needlang AS ( + SELECT man FROM lst GROUP BY man HAVING count(*) > 1 + ), needhash AS ( + SELECT man, locale FROM lst GROUP BY man, locale HAVING count(*) > 1 + ) SELECT name, section, shorthash, filename, locale + , EXISTS(SELECT 1 FROM needlang WHERE man = l.man) AS needlang + , EXISTS(SELECT 1 FROM needhash WHERE man = l.man AND locale = l.locale) AS needhash + FROM lst l + ORDER BY name, section, locale, filename + '); + + # Latest version of this package determines last modification date of the page. + tuwf->resLastMod($vers->[0]{released}); + + my $subtitle = " / $pkg->{category} / $pkg->{name}"; + my $pkgpath = "$sys->{short}/$pkg->{category}/$pkg->{name}"; + framework_ title => "$sys->{full}$subtitle $sel->{version}", mainclass => 'pkgpage', sub { + h1_ sub { + a_ href => "/pkg/$sys->{short}", $sys->{full}; + txt_ $subtitle; + }; + + div_ sub { + section_ sub { + h2_ 'Versions'; + ul_ sub { + li_ sub { + a_ href => "/pkg/$pkgpath/$_->{version}", $_->{version} if $_ != $sel; + b_ " $_->{version}" if $_ == $sel; + small_ " $_->{released}"; + } for(@$vers); + } + }; + + section_ sub { + h2_ "Manuals for version $sel->{version}"; + paginate_ "/pkg/$pkgpath/$sel->{version}?p=", $count, 200, $p; + ul_ sub { + li_ sub { + # Only add the hash or locale to the URL if it's necessary to select the proper man page. + my $ext = $_->{needhash} ? '.'.shorthash_to_hex $_->{shorthash} : $_->{needlang} && length $_->{locale} ? ".$_->{locale}" : ''; + a_ href => "/man$ext/$pkgpath/$sel->{version}/$_->{name}.$_->{section}", "$_->{name}($_->{section})"; + b_ " $_->{locale}" if $_->{locale}; + small_ " $_->{filename}"; + } for(@$mans); + }; + paginate_ "/pkg/$pkgpath/$sel->{version}?p=", $count, 200, $p; + }; + }; + } +}; -sub dbSystemGet { - return shift->dbAll('SELECT id, name, release, short, relorder FROM systems ORDER BY name, relorder'); -} +# /browse/<pkg> has been moved to /pkg/ with the package category added to the path +TUWF::get qr{/browse/([^/]+)} => sub { tuwf->resRedirect('/pkg/'.tuwf->capture(1), 'perm') }; +TUWF::get qr{/browse/([^/]+)/([^/]+)(?:/([^/]+))?} => sub { + my($sys, $name, $ver) = tuwf->captures(1,2,3); + $sys = sysbyshort->{$sys}; + return tuwf->resNotFound if !$sys; + my $pkgs = tuwf->dbRowi('SELECT category FROM packages WHERE system =', \$sys->{id}, 'AND name =', \$name, 'LIMIT 1'); + return tuwf->resNotFound if !defined $pkgs->{category}; + tuwf->resRedirect("/pkg/$sys->{short}/$pkgs->{category}/$name".($ver ? "/$ver" :''), 'perm'); +}; -# Options: sysid char hasman page results countonly -sub dbPackageGet { - my $s = shift; - my %o = (results => 10, page => 1, @_); - - my @where = ( - $o{sysid} ? ('system = ?' => $o{sysid} ) : (), - $o{category} ? ('category = ?' => $o{category}) : (), - $o{name} ? ('name = ?' => $o{name} ) : (), - # This seems slow, perhaps cache? - defined($o{hasman}) ? ('!s EXISTS(SELECT 1 FROM package_versions pv WHERE pv.package = p.id AND EXISTS(SELECT 1 FROM man m WHERE m.package = pv.id))' => $o{hasman}?'':'NOT') : (), - $o{char} ? ( 'LOWER(SUBSTR(name, 1, 1)) = ?' => $o{char} ) : (), - defined($o{char}) && !$o{char} ? ( '(ASCII(name) < 97 OR ASCII(name) > 122) AND (ASCII(name) < 65 OR ASCII(name) > 90)' => 1 ) : (), - ); - - my $select = $o{countonly} ? 'COUNT(*) as count' : 'id, system, name, category'; - my $order = $o{countonly} ? '' : 'ORDER BY name'; - - my($r, $np) = $s->dbPage(\%o, - 'SELECT !s FROM packages p !W !s', - $select, \@where, $order - ); - wantarray ? ($r, $np) : $r; -} +# Redirect for the system selection box, for visitors who have disabled JS. +TUWF::get qr{/sysredir/([^/]+)} => sub { tuwf->resRedirect('/man/'.(tuwf->reqGet('system')//'arch').'/'.tuwf->capture(1), 'temp') }; + +# Redirect for a specific language for a man page. I have no idea if anyone +# still uses this URL format, but it was supported at some point, so let's keep +# it around. +TUWF::get qr{/lang/([^/]+)/([^/]+)} => sub { tuwf->resRedirect('/man.'.tuwf->capture(1).'/'.tuwf->capture(2), 'temp') }; + + +TUWF::get '/json/tree.json' => sub { + my $f = tuwf->validate(get => + name => { default => '', maxlength => 256 }, + section => { default => '', maxlength => 32 }, + locale => { default => sub{$_[0]}, maxlength => 32 }, + cur => { default => '', regex => qr/^[a-fA-F0-9]{40}$/ }, + hash => { default => '', regex => qr/^[a-fA-F0-9]{40}$/ }, + )->data; + return tuwf->resNotFound() if !$f->{hash} && !($f->{section} && $f->{name}); + + my $l = tuwf->dbAlli(" + SELECT p.system, p.category, p.name AS package, v.version, v.released, v.id AS verid, m.name, m.section, f.filename, f.shorthash, l.locale + FROM files f + JOIN locales l ON l.id = f.locale + JOIN mans m ON m.id = f.man + JOIN package_versions v ON v.id = f.pkgver + JOIN packages p ON p.id = v.package + JOIN systems s ON s.id = p.system + WHERE", sql_and( + length $f->{hash} ? sql 'f.content = (SELECT id FROM contents WHERE hash = decode(', \$f->{hash}, ", 'hex'))" : (), + length $f->{name} ? sql 'm.name =', \$f->{name} : (), + length $f->{section} ? sql 'm.section =', \$f->{section} : (), + defined $f->{locale} ? sql 'l.locale =', \$f->{locale} : (), + ), ' + ORDER BY s.name, s.id DESC, p.name, v.released DESC, m.name, l.locale, f.filename + '); + + # Convert the list into a tree + my $cur = $f->{cur} ? shorthash_to_int substr $f->{cur}, 0, 8 : 0; + my $tree = []; + my($sys, $sysver, $pkg, $pkgver); + for my $m (@$l) { + my $sysname = sysbyid->{$m->{system}}{name}; + if(!$sys || $sysname ne $sys->{name}) { + $sys = { name => $sysname, childs => [] }; + $sysver = undef; + push @$tree, $sys; + } + my $sysversion = sysbyid->{$m->{system}}{release} || ''; + if(!$sysver || $sysversion ne $sysver->{name}) { + $sysver = { name => $sysversion, childs => [] }; + $pkg = undef; + push @{$sys->{childs}}, $sysver; + } -sub dbPackageVersions { - my($s, $id, $version) = @_; + if(!$pkg || $m->{package} ne $pkg->{name}) { + $pkg = { name => $m->{package}, i => $m->{category}, table => [] }; + $pkgver = undef; + push @{$sysver->{childs}}, $pkg; + } - my %where = ( - 'package = ?' => $id, - $version ? ('version = ?' => $version) : (), - 'EXISTS(SELECT 1 FROM man m WHERE m.package = v.id)' => 1, - ); + push @{$pkg->{table}}, [ + $pkgver && $pkgver eq $m->{version} ? {name=>''} : + {name => $m->{version}, href => "/pkg/".sysbyid->{$m->{system}}{short}."/$m->{category}/$m->{package}/$m->{version}"}, + { name => "$m->{name}($m->{section})", + $f->{hash} || $cur == $m->{shorthash} ? () + : (href => sprintf('/%s/%s', $m->{name}, shorthash_to_hex $m->{shorthash})) + }, + { name => shorthash_to_hex($m->{shorthash}), + $f->{hash} || $cur == $m->{shorthash} ? () + : (href => sprintf('/%s/%s', $m->{name}, shorthash_to_hex $m->{shorthash})) + }, + { name => $m->{filename} } + ]; + $pkgver = $m->{version}; + } - return $s->dbAll(q{ - SELECT id, version, released - FROM package_versions v !W - ORDER BY released DESC}, - \%where) -} + # Determine which elements to show/hide by default. + # It might make more sense to do this in JS, but since I am utterly + # incapable of writing maintainable JS I'm doing it here in order to keep the + # JS stupid and simple. + # TODO: Highlight systems/packages where the 'current' man page is? + for my $sys (@$tree) { + $sys->{expand} = 1 if $sys->{childs}[0]{name}; # Expand all systems that have named versions + $sys->{expand} = 1 if $f->{hash}; # Expand everything on 'location' + + my $i = 0; + for my $sysver (@{$sys->{childs}}) { + $i++; + $sysver->{expand} = 1 if !$sysver->{name}; # Expand unnamed versions (since you can't click them) + $sysver->{expand} = 1 if $f->{hash}; # Expand everything on 'location' + $sysver->{hide} = 1 if $i > 3 && @{$sys->{childs}} > 5; # Show only the first 3 versions + + for my $pkg (@{$sysver->{childs}}) { + $pkg->{expand} = 1 if @{$sysver->{childs}} <= 3; # Expand everything if there's not too many things to expand + $pkg->{expand} = 1 if $f->{hash}; # Expand everything on 'location' + + # TODO: Show/Hide duplicate hashes? + } + } + } + tuwf->resJSON($tree); +}; -sub dbStats { - return $_[0]->dbRow('SELECT * FROM stats_cache'); -} +TUWF::run(); diff --git a/www/man.css b/www/man.css index 3e86d9f..a1dbe15 100644 --- a/www/man.css +++ b/www/man.css @@ -1,84 +1,54 @@ -/* TODO: column-width is better done in Perl? (More like a column-count, then) */ - -* { margin: 0; padding: 0; font-family: "Trebuchet MS", sans-serif; } -html { background: #333; padding: 0 10px; } -body { margin: 10px auto 50px auto; max-width: 1100px; border-collapse: separate; padding-bottom: 10px; border-radius: 10px; box-shadow: 0 10px 10px #def; } -h1 { font-size: 24px; font-weight: normal; color: #abc; } -h2 { font-size: 21px; margin-top: 40px; color: #468; font-weight: normal; clear: left } -h2 + i { font-size: 12px; } -h3 { font-size: 16px; margin-top: 20px; color: #468; font-weight: normal } -a { color: #048; } +* { margin: 0; padding: 0; font-size: inherit; font-family: "Trebuchet MS", sans-serif; } +html { background: #fff; padding: 0 10px; color: #000 } +body { margin: 10px auto 50px auto; max-width: 1100px; border-collapse: separate; padding-bottom: 10px } +h1 { font-size: 24px; font-weight: normal } +h2 { font-size: 21px; margin-top: 35px; margin-bottom: 7px; font-weight: normal; border-bottom: 1px solid #ccc } +h3 { font-size: 18px; margin-top: 20px; margin-bottom: 5px; font-weight: normal } +a { color: #048; text-decoration: none } a:hover { text-decoration: underline; color: #48B;} -code { font-family: "Lucida Console", Monospace; font-size: 12px; background-color: #f0f8ff } +p { margin-bottom: 15px } +dl { margin-bottom: 15px } +dt { margin: 0 0 5px 0 } +dd { margin: 0 0 10px 10px; } +ul { margin: 0 0 15px 20px } +input { font-size: 0.9em; padding: 1px 2px } +code { font-family: "Lucida Console", Monospace; font-size: 12px; background-color: #f0f8ff; padding: 1px } +small { color: #aaa } .hidden { display: none!important; } -#header { padding: 4px 20px; border-bottom: 1px solid #888; font: 24px "Arial"; background: url('images/gradients.png') repeat-x; border-radius: 8px 8px 0 0; } -#header a { color: #f8f8f8; text-decoration: none; font-weight: bold; } -#header a:hover { background: none; } -#header form { float: right; } -#header input[type=text] { color: #000; width: 260px; padding: 1px 2px 1px 15px; border-radius: 12px 0 0 12px; border: 1px solid #444; - box-shadow: 1px 1px 3px #fff, -1px -1px 2px #234; background: url('images/gradients.png') 0 -105px repeat-x; height: 15px; } -#header input[type=text]:hover, #header input[type=text]:focus { background: url('images/gradients.png') 0 -122px repeat-x; outline: none; } -#header input[type=submit] { height: 23px; width: 62px; background-image: url('images/search.png'); border: 0; margin: 0 0 0 -5px; cursor: pointer } - -#body { padding: 10px 10px 20px 10px; background: #fff } - -#systems a, -.charselect a, -.paginate a { color: #048; font-family: "Verdana"; font-weight: normal; text-decoration: none; padding: 3px 5px; border-radius: 4px; } - -.charselect b, -.paginate b { font-family: "Verdana"; padding: 3px } - -#systems a:hover, -.charselect a:hover, -.paginate a:hover { background: #cde; } - -p.txt { margin: 5px 0 0 10px; max-width: 700px } -#external { list-style-type: none; margin-left: 10px } - -i.grayedout { color: #aaa; font-size: 13px; } - -#about { max-width: 700px } -#about h2 { margin-bottom: 5px } -#about dl { margin-bottom: 5px } -#about dt { margin: 5px 0 0 10px } -#about p { margin: 0 0 0 10px; } -#about dd { margin: 0 0 0 20px; } -#about ul { padding: 5px 0 5px 30px } - -#systems li { display: block; float: left; width: 300px; min-height: 80px; margin: 15px 10px; padding-left: 60px } -#systems span { display: block; margin-left: -55px; float: left; width: 50px; height: 50px; background-repeat: no-repeat; } -#systems b { font-size: 24px; display: block } - -#pkglist .charselect { float: right } -#pkglist .paginate { display: block; margin: 10px; width: 100%; text-align: center } - -#packages { margin: 20px 0; -webkit-column-width: 300px; -moz-column-width: 300px; column-width: 300px } -#packages li { display: block; } -#packages i { color: #aaa; font-size: 13px; } -#packages a { padding-right: 2px } - -#searchres { margin: 20px 0; -webkit-column-width: 300px; -moz-column-width: 300px; column-width: 300px } -#searchres li { display: block; } -#searchres i { color: #aaa; font-size: 13px; } -#searchres a { padding-right: 2px } - -#pkgversions { margin-top: 10px } -#pkgversions h2 { margin: 0 } -#pkgversions { float: left; padding: 0 10px; } -#pkgversions ul { margin: 10px 0; padding-right: 10px; border-right: 1px dashed #468 } -#pkgversions li { display: block; } -#pkgversions i { color: #aaa; font-size: 13px; } -#pkgversions a { padding-right: 2px } - -#pkgmans { margin-top: 10px } -#pkgmans h2 { margin: 0 } -#pkgmans { float: left } -#pkgmans .paginate { margin: 10px 0 } -#pkgmans ul { margin: 10px 0 } -#pkgmans li { display: block } -#pkgmans i { color: #aaa; font-size: 13px; } +header { border-bottom: 3px dotted #ccc; display: flex; justify-content: space-between; align-items: end; flex-wrap: wrap } +header a { font: 24px "Arial", serif; font-weight: bold } +header form { padding-bottom: 3px } +header input[type=text] { width: 100px } + +footer { border-top: 3px dotted #ccc; color: #666; font-size: 12px; display: flex; justify-content: space-between; flex-wrap: wrap } + +main { padding: 10px 0 25px 0 } +main.thin { max-width: 700px; margin: 0 auto } + +.systems > div { margin: 25px 0; display: flex; align-items: start } +.systems > div > a { display: flex; align-items: center } +.systems img { width: 50px; height: 50px; margin-right: 10px } +.systems b { font-size: 24px; display: block } +.systems > div > div > a { padding-right: 10px } + +.searchres ul { margin: 20px 0; list-style-type: none; column-width: 300px } +.searchres a { padding-right: 2px } + +.charselect a, .paginate a { display: inline-block; padding: 3px 5px } +.charselect b, .paginate b { display: inline-block; padding: 3px 5px } + +.pkglist > div { display: flex; align-items: start; justify-content: space-between; flex-wrap: wrap } +.pkglist .paginate { margin: 10px 0; text-align: center } +.pkglist ul { margin: 20px 0; list-style-type: none; column-width: 300px } +.pkglist ul a { padding-right: 2px } + +.pkgpage > div { display: flex; flex-wrap: wrap; margin-top: 15px } +.pkgpage h2 { border-bottom: 0; margin: 0 0 10px 0 } +.pkgpage ul { margin: 0; list-style-type: none } +.pkgpage section:first-child { border-right: 1px dashed #468; padding: 0 10px } +.pkgpage section:last-child { padding: 0 10px } + #manbuttons h1 { display: inline; margin: 0 20px 0 0; vertical-align: middle } #manbuttons ul { list-style-type: none; display: inline-block } @@ -91,7 +61,7 @@ i.grayedout { color: #aaa; font-size: 13px; } #manres { margin: 0 0 10px 0; width: 70%; padding: 10px; box-sizing: border-box; background: #f0f8ff; border-radius: 10px; border-left: 1px dashed #333; border-right: 1px dashed #333 } #manres i { color: #aaa; font-size: 13px; margin-left: 7px } -#manres ul { list-style-type: none } +#manres ul { list-style-type: none; margin-bottom: 0 } #manres ul a { outline: none; text-decoration: none } #manres ul .oldver a { color: #aaa; font-size: 13px } #manres div > ul { margin-top: 5px } /* System names */ @@ -106,21 +76,21 @@ i.grayedout { color: #aaa; font-size: 13px; } #manres table { margin: 2px 10px; } #manres table tr td:nth-child(1) { min-width: 80px } -#nav { background: #f0f8ff; color: #036; float: right; padding: 8px; width: 250px; margin-bottom: 10px; border-radius: 8px; } -#nav b { text-transform: uppercase; font-size: 13px } -#nav p { margin: 3px 5px 20px 5px } -#nav p a, #nav p i { padding: 3px 5px; font-size: 13px; font-style: normal; text-decoration: none;} -#nav p a:hover, #nav p i { background: #cde; border-radius: 5px } -#nav ul { list-style-type: none; margin: 3px 10px 0 20px } -#nav ul li a { overflow: hidden; margin-left: -10px; text-decoration: none; text-transform: capitalize } - -#contents { margin: 10px 0 0 0 } - -#footer { height: 60px; clear: both; padding: 4px 10px; color: #f8f8f8; margin: 0 0 -20px 0; - border-top: 1px solid #888; font-size: 13px; background: url('images/gradients.png') 0 -37px repeat-x; border-radius: 0 0 8px 8px; } -#footer a { font-size: 13px; padding: 0; color: #f8f8f8; } -#footer a:hover { background: none; } - +.manpage nav { background: #f0f8ff; color: #036; float: right; padding: 8px; width: 250px; margin-bottom: 10px; border-radius: 8px; } +.manpage nav b { text-transform: uppercase; font-size: 13px } +.manpage nav p { margin: 3px 5px 20px 5px } +.manpage nav p a, +.manpage nav p i { padding: 3px 5px; font-size: 13px; font-style: normal; text-decoration: none;} +.manpage nav p a:hover, +.manpage nav p i { background: #cde; border-radius: 5px } +.manpage nav ul { list-style-type: none; margin: 3px 10px 0 20px } +.manpage nav ul li a { overflow: hidden; margin-left: -10px; text-decoration: none; text-transform: capitalize } +.manpage nav form { margin: 0 0 10px 0 } +.manpage nav select { width: 200px } +.manpage nav input { width: 40px } + + +pre { margin: 10px 0 0 0 } pre, pre * { font-family: "Lucida Console", Monospace; font-size: 15px } pre b, pre em, pre a { color: #369; font-weight: normal; text-decoration: none } pre em { font-style: italic } @@ -454,29 +454,8 @@ function dsResults(hr, obj) { )); ul.appendChild(tag('li', tag('a', {href:'#', onclick: buttonclick, - 'data-url': '/json/tree.json?hash='+hash+';name='+name+';section='+section, + 'data-url': '/json/tree.json?hash='+hash, 'data-p': 'This manual page was found in the following locations.'}, 'locations'))); })(); })(); - - - - -// The "more..." links on the homepage. -(function(){ - var sys = byId('systems'); - if(!sys) - return; - var f = function() { - var l = byName(this.parentNode, 'a'); - var show = hasClass(l[3], 'hidden'); - for(var i=3; i<l.length-1; i++) - setClass(l[i], 'hidden', !show); - setText(this, show ? '...less' : 'more...'); - return false - }; - var l = byClass(sys, 'a', 'more'); - for(var i=0; i<l.length; i++) - l[i].onclick = f; -})(); |
