diff --git a/mosesdecoder/COPYING b/mosesdecoder/COPYING new file mode 100644 index 0000000000000000000000000000000000000000..805dbfe1232acbb6d5eee915e477ffd17193cc66 --- /dev/null +++ b/mosesdecoder/COPYING @@ -0,0 +1,460 @@ + + GNU LESSER GENERAL PUBLIC LICENSE + Version 2.1, February 1999 + + Copyright (C) 1991, 1999 Free Software Foundation, Inc. + 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + +[This is the first released version of the Lesser GPL. It also counts + as the successor of the GNU Library Public License, version 2, hence + the version number 2.1.] + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +Licenses are intended to guarantee your freedom to share and change +free software--to make sure the software is free for all its users. + + This license, the Lesser General Public License, applies to some +specially designated software packages--typically libraries--of the +Free Software Foundation and other authors who decide to use it. You +can use it too, but we suggest you first think carefully about whether +this license or the ordinary General Public License is the better +strategy to use in any particular case, based on the explanations +below. + + When we speak of free software, we are referring to freedom of use, +not price. Our General Public Licenses are designed to make sure that +you have the freedom to distribute copies of free software (and charge +for this service if you wish); that you receive source code or can get +it if you want it; that you can change the software and use pieces of +it in new free programs; and that you are informed that you can do +these things. + + To protect your rights, we need to make restrictions that forbid +distributors to deny you these rights or to ask you to surrender these +rights. These restrictions translate to certain responsibilities for +you if you distribute copies of the library or if you modify it. + + For example, if you distribute copies of the library, whether gratis +or for a fee, you must give the recipients all the rights that we gave +you. You must make sure that they, too, receive or can get the source +code. If you link other code with the library, you must provide +complete object files to the recipients, so that they can relink them +with the library after making changes to the library and recompiling +it. And you must show them these terms so they know their rights. + + We protect your rights with a two-step method: (1) we copyright the +library, and (2) we offer you this license, which gives you legal +permission to copy, distribute and/or modify the library. + + To protect each distributor, we want to make it very clear that +there is no warranty for the free library. Also, if the library is +modified by someone else and passed on, the recipients should know +that what they have is not the original version, so that the original +author's reputation will not be affected by problems that might be +introduced by others. + + Finally, software patents pose a constant threat to the existence of +any free program. We wish to make sure that a company cannot +effectively restrict the users of a free program by obtaining a +restrictive license from a patent holder. Therefore, we insist that +any patent license obtained for a version of the library must be +consistent with the full freedom of use specified in this license. + + Most GNU software, including some libraries, is covered by the +ordinary GNU General Public License. This license, the GNU Lesser +General Public License, applies to certain designated libraries, and +is quite different from the ordinary General Public License. We use +this license for certain libraries in order to permit linking those +libraries into non-free programs. + + When a program is linked with a library, whether statically or using +a shared library, the combination of the two is legally speaking a +combined work, a derivative of the original library. The ordinary +General Public License therefore permits such linking only if the +entire combination fits its criteria of freedom. The Lesser General +Public License permits more lax criteria for linking other code with +the library. + + We call this license the "Lesser" General Public License because it +does Less to protect the user's freedom than the ordinary General +Public License. It also provides other free software developers Less +of an advantage over competing non-free programs. These disadvantages +are the reason we use the ordinary General Public License for many +libraries. However, the Lesser license provides advantages in certain +special circumstances. + + For example, on rare occasions, there may be a special need to +encourage the widest possible use of a certain library, so that it +becomes a de-facto standard. To achieve this, non-free programs must +be allowed to use the library. A more frequent case is that a free +library does the same job as widely used non-free libraries. In this +case, there is little to gain by limiting the free library to free +software only, so we use the Lesser General Public License. + + In other cases, permission to use a particular library in non-free +programs enables a greater number of people to use a large body of +free software. For example, permission to use the GNU C Library in +non-free programs enables many more people to use the whole GNU +operating system, as well as its variant, the GNU/Linux operating +system. + + Although the Lesser General Public License is Less protective of the +users' freedom, it does ensure that the user of a program that is +linked with the Library has the freedom and the wherewithal to run +that program using a modified version of the Library. + + The precise terms and conditions for copying, distribution and +modification follow. Pay close attention to the difference between a +"work based on the library" and a "work that uses the library". The +former contains code derived from the library, whereas the latter must +be combined with the library in order to run. + + GNU LESSER GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License Agreement applies to any software library or other +program which contains a notice placed by the copyright holder or +other authorized party saying it may be distributed under the terms of +this Lesser General Public License (also called "this License"). +Each licensee is addressed as "you". + + A "library" means a collection of software functions and/or data +prepared so as to be conveniently linked with application programs +(which use some of those functions and data) to form executables. + + The "Library", below, refers to any such software library or work +which has been distributed under these terms. A "work based on the +Library" means either the Library or any derivative work under +copyright law: that is to say, a work containing the Library or a +portion of it, either verbatim or with modifications and/or translated +straightforwardly into another language. (Hereinafter, translation is +included without limitation in the term "modification".) + + "Source code" for a work means the preferred form of the work for +making modifications to it. For a library, complete source code means +all the source code for all modules it contains, plus any associated +interface definition files, plus the scripts used to control +compilation and installation of the library. + + Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running a program using the Library is not restricted, and output from +such a program is covered only if its contents constitute a work based +on the Library (independent of the use of the Library in a tool for +writing it). Whether that is true depends on what the Library does +and what the program that uses the Library does. + + 1. You may copy and distribute verbatim copies of the Library's +complete source code as you receive it, in any medium, provided that +you conspicuously and appropriately publish on each copy an +appropriate copyright notice and disclaimer of warranty; keep intact +all the notices that refer to this License and to the absence of any +warranty; and distribute a copy of this License along with the +Library. + + You may charge a fee for the physical act of transferring a copy, +and you may at your option offer warranty protection in exchange for a +fee. + + 2. You may modify your copy or copies of the Library or any portion +of it, thus forming a work based on the Library, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) The modified work must itself be a software library. + + b) You must cause the files modified to carry prominent notices + stating that you changed the files and the date of any change. + + c) You must cause the whole of the work to be licensed at no + charge to all third parties under the terms of this License. + + d) If a facility in the modified Library refers to a function or a + table of data to be supplied by an application program that uses + the facility, other than as an argument passed when the facility + is invoked, then you must make a good faith effort to ensure that, + in the event an application does not supply such function or + table, the facility still operates, and performs whatever part of + its purpose remains meaningful. + + (For example, a function in a library to compute square roots has + a purpose that is entirely well-defined independent of the + application. Therefore, Subsection 2d requires that any + application-supplied function or table used by this function must + be optional: if the application does not supply it, the square + root function must still compute square roots.) + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Library, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Library, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote +it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Library. + +In addition, mere aggregation of another work not based on the Library +with the Library (or with a work based on the Library) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may opt to apply the terms of the ordinary GNU General Public +License instead of this License to a given copy of the Library. To do +this, you must alter all the notices that refer to this License, so +that they refer to the ordinary GNU General Public License, version 2, +instead of to this License. (If a newer version than version 2 of the +ordinary GNU General Public License has appeared, then you can specify +that version instead if you wish.) Do not make any other change in +these notices. + + Once this change is made in a given copy, it is irreversible for +that copy, so the ordinary GNU General Public License applies to all +subsequent copies and derivative works made from that copy. + + This option is useful when you wish to copy part of the code of +the Library into a program that is not a library. + + 4. You may copy and distribute the Library (or a portion or +derivative of it, under Section 2) in object code or executable form +under the terms of Sections 1 and 2 above provided that you accompany +it with the complete corresponding machine-readable source code, which +must be distributed under the terms of Sections 1 and 2 above on a +medium customarily used for software interchange. + + If distribution of object code is made by offering access to copy +from a designated place, then offering equivalent access to copy the +source code from the same place satisfies the requirement to +distribute the source code, even though third parties are not +compelled to copy the source along with the object code. + + 5. A program that contains no derivative of any portion of the +Library, but is designed to work with the Library by being compiled or +linked with it, is called a "work that uses the Library". Such a +work, in isolation, is not a derivative work of the Library, and +therefore falls outside the scope of this License. + + However, linking a "work that uses the Library" with the Library +creates an executable that is a derivative of the Library (because it +contains portions of the Library), rather than a "work that uses the +library". The executable is therefore covered by this License. +Section 6 states terms for distribution of such executables. + + When a "work that uses the Library" uses material from a header file +that is part of the Library, the object code for the work may be a +derivative work of the Library even though the source code is not. +Whether this is true is especially significant if the work can be +linked without the Library, or if the work is itself a library. The +threshold for this to be true is not precisely defined by law. + + If such an object file uses only numerical parameters, data +structure layouts and accessors, and small macros and small inline +functions (ten lines or less in length), then the use of the object +file is unrestricted, regardless of whether it is legally a derivative +work. (Executables containing this object code plus portions of the +Library will still fall under Section 6.) + + Otherwise, if the work is a derivative of the Library, you may +distribute the object code for the work under the terms of Section 6. +Any executables containing that work also fall under Section 6, +whether or not they are linked directly with the Library itself. + + 6. As an exception to the Sections above, you may also combine or +link a "work that uses the Library" with the Library to produce a +work containing portions of the Library, and distribute that work +under terms of your choice, provided that the terms permit +modification of the work for the customer's own use and reverse +engineering for debugging such modifications. + + You must give prominent notice with each copy of the work that the +Library is used in it and that the Library and its use are covered by +this License. You must supply a copy of this License. If the work +during execution displays copyright notices, you must include the +copyright notice for the Library among them, as well as a reference +directing the user to the copy of this License. Also, you must do one +of these things: + + a) Accompany the work with the complete corresponding + machine-readable source code for the Library including whatever + changes were used in the work (which must be distributed under + Sections 1 and 2 above); and, if the work is an executable linked + with the Library, with the complete machine-readable "work that + uses the Library", as object code and/or source code, so that the + user can modify the Library and then relink to produce a modified + executable containing the modified Library. (It is understood + that the user who changes the contents of definitions files in the + Library will not necessarily be able to recompile the application + to use the modified definitions.) + + b) Use a suitable shared library mechanism for linking with the + Library. A suitable mechanism is one that (1) uses at run time a + copy of the library already present on the user's computer system, + rather than copying library functions into the executable, and (2) + will operate properly with a modified version of the library, if + the user installs one, as long as the modified version is + interface-compatible with the version that the work was made with. + + c) Accompany the work with a written offer, valid for at least + three years, to give the same user the materials specified in + Subsection 6a, above, for a charge no more than the cost of + performing this distribution. + + d) If distribution of the work is made by offering access to copy + from a designated place, offer equivalent access to copy the above + specified materials from the same place. + + e) Verify that the user has already received a copy of these + materials or that you have already sent this user a copy. + + For an executable, the required form of the "work that uses the +Library" must include any data and utility programs needed for +reproducing the executable from it. However, as a special exception, +the materials to be distributed need not include anything that is +normally distributed (in either source or binary form) with the major +components (compiler, kernel, and so on) of the operating system on +which the executable runs, unless that component itself accompanies +the executable. + + It may happen that this requirement contradicts the license +restrictions of other proprietary libraries that do not normally +accompany the operating system. Such a contradiction means you cannot +use both them and the Library together in an executable that you +distribute. + + 7. You may place library facilities that are a work based on the +Library side-by-side in a single library together with other library +facilities not covered by this License, and distribute such a combined +library, provided that the separate distribution of the work based on +the Library and of the other library facilities is otherwise +permitted, and provided that you do these two things: + + a) Accompany the combined library with a copy of the same work + based on the Library, uncombined with any other library + facilities. This must be distributed under the terms of the + Sections above. + + b) Give prominent notice with the combined library of the fact + that part of it is a work based on the Library, and explaining + where to find the accompanying uncombined form of the same work. + + 8. You may not copy, modify, sublicense, link with, or distribute +the Library except as expressly provided under this License. Any +attempt otherwise to copy, modify, sublicense, link with, or +distribute the Library is void, and will automatically terminate your +rights under this License. However, parties who have received copies, +or rights, from you under this License will not have their licenses +terminated so long as such parties remain in full compliance. + + 9. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Library or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Library (or any work based on the +Library), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Library or works based on it. + + 10. Each time you redistribute the Library (or any work based on the +Library), the recipient automatically receives a license from the +original licensor to copy, distribute, link with or modify the Library +subject to these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties with +this License. + + 11. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Library at all. For example, if a patent +license would not permit royalty-free redistribution of the Library by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Library. + +If any portion of this section is held invalid or unenforceable under +any particular circumstance, the balance of the section is intended to +apply, and the section as a whole is intended to apply in other +circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + 12. If the distribution and/or use of the Library is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Library under this License +may add an explicit geographical distribution limitation excluding those +countries, so that distribution is permitted only in or among +countries not thus excluded. In such case, this License incorporates +the limitation as if written in the body of this License. + + 13. The Free Software Foundation may publish revised and/or new +versions of the Lesser General Public License from time to time. +Such new versions will be similar in spirit to the present version, +but may differ in detail to address new problems or concerns. + +Each version is given a distinguishing version number. If the Library +specifies a version number of this License which applies to it and +"any later version", you have the option of following the terms and +conditions either of that version or of any later version published by +the Free Software Foundation. If the Library does not specify a +license version number, you may choose any version ever published by +the Free Software Foundation. + + 14. If you wish to incorporate parts of the Library into other free +programs whose distribution conditions are incompatible with these, +write to the author to ask for permission. For software which is +copyrighted by the Free Software Foundation, write to the Free +Software Foundation; we sometimes make exceptions for this. Our +decision will be guided by the two goals of preserving the free status +of all derivatives of our free software and of promoting the sharing +and reuse of software generally. + + NO WARRANTY + + 15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO +WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW. +EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR +OTHER PARTIES PROVIDE THE LIBRARY "AS IS" WITHOUT WARRANTY OF ANY +KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE +LIBRARY IS WITH YOU. SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME +THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN +WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY +AND/OR REDISTRIBUTE THE LIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU +FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR +CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE +LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING +RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A +FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF +SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH +DAMAGES. + diff --git a/mosesdecoder/bjam b/mosesdecoder/bjam new file mode 100644 index 0000000000000000000000000000000000000000..0ebf105c373e04a5ca1d12f79f8ae625af44db51 --- /dev/null +++ b/mosesdecoder/bjam @@ -0,0 +1,23 @@ +#!/bin/bash +set -e +top="$(dirname "$0")" +if + bjam="$(which bjam 2>/dev/null)" && #exists + [ ${#bjam} != 0 ] && #paranoia about which printing nothing then returning true + ! grep UFIHGUFIHBDJKNCFZXAEVA "${bjam}" /dev/null && #bjam in path isn't this script + "${bjam}" --sanity-test 2>/dev/null |grep Sane >/dev/null && #The test in jam-files/sanity.jam passes + (cd "${top}/jam-files/fail" && ! "${bjam}") >/dev/null #Returns non-zero on failure +then + #Delegate to system bjam + exec "${bjam}" "$@" +fi + +if [ ! -x "$top"/jam-files/bjam ] || "$top"/jam-files/bjam -v |grep 2011.4 >/dev/null; then + pushd "$top/jam-files/engine" + ./build.sh + cp -f bin.*/bjam ../bjam + popd +fi + +export BOOST_BUILD_PATH="$top"/jam-files/boost-build +exec "$top"/jam-files/bjam "$@" diff --git a/mosesdecoder/moses2/AlignmentInfoCollection.h b/mosesdecoder/moses2/AlignmentInfoCollection.h new file mode 100644 index 0000000000000000000000000000000000000000..0d409430d76cc87fb60bab7f660d21a919e58b1a --- /dev/null +++ b/mosesdecoder/moses2/AlignmentInfoCollection.h @@ -0,0 +1,81 @@ +/*********************************************************************** + Moses - statistical machine translation system + Copyright (C) 2006-2011 University of Edinburgh + + This library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + This library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with this library; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +***********************************************************************/ + +#pragma once + +#include "AlignmentInfo.h" + +#include + +#ifdef WITH_THREADS +#include +#include +#endif + +namespace Moses2 +{ + +/** Singleton collection of all AlignmentInfo objects. + * Used as a cache of all alignment info to save space. + */ +class AlignmentInfoCollection +{ +public: + static AlignmentInfoCollection &Instance() { + return s_instance; + } + + /** Returns a pointer to an AlignmentInfo object with the same source-target + * alignment pairs as given in the argument. If the collection already + * contains such an object then returns a pointer to it; otherwise a new + * one is inserted. + */ +private: + const AlignmentInfo* Add(AlignmentInfo const& ainfo); + +public: + template + AlignmentInfo const * + Add(ALNREP const & aln) { + return this->Add(AlignmentInfo(aln)); + } + + //! Returns a pointer to an empty AlignmentInfo object. + const AlignmentInfo &GetEmptyAlignmentInfo() const; + +private: + typedef std::set AlignmentInfoSet; + + + //! Only a single static variable should be created. + AlignmentInfoCollection(); + ~AlignmentInfoCollection(); + + static AlignmentInfoCollection s_instance; + +#ifdef WITH_THREADS + //reader-writer lock + mutable boost::shared_mutex m_accessLock; +#endif + + AlignmentInfoSet m_collection; + const AlignmentInfo *m_emptyAlignmentInfo; +}; + +} diff --git a/mosesdecoder/moses2/ArcLists.h b/mosesdecoder/moses2/ArcLists.h new file mode 100644 index 0000000000000000000000000000000000000000..b0269d8d87a9b11db5aa186f273107204e09844b --- /dev/null +++ b/mosesdecoder/moses2/ArcLists.h @@ -0,0 +1,44 @@ +/* + * ArcList.h + * + * Created on: 26 Oct 2015 + * Author: hieu + */ +#pragma once +#include +#include +#include + +namespace Moses2 +{ +class System; + +class HypothesisBase; + +typedef std::vector ArcList; + +class ArcLists +{ +public: + ArcLists(); + virtual ~ArcLists(); + + void AddArc(bool added, const HypothesisBase *currHypo, + const HypothesisBase *otherHypo); + void Sort(); + void Delete(const HypothesisBase *hypo); + + const ArcList &GetArcList(const HypothesisBase *hypo) const; + + std::string Debug(const System &system) const; +protected: + typedef std::unordered_map Coll; + Coll m_coll; + + ArcList &GetArcList(const HypothesisBase *hypo); + ArcList &GetAndDetachArcList(const HypothesisBase *hypo); + +}; + +} + diff --git a/mosesdecoder/moses2/EstimatedScores.cpp b/mosesdecoder/moses2/EstimatedScores.cpp new file mode 100644 index 0000000000000000000000000000000000000000..e71647ce529df13cd503675af1be2c02c72eacdd --- /dev/null +++ b/mosesdecoder/moses2/EstimatedScores.cpp @@ -0,0 +1,117 @@ +// $Id$ +// vim:tabstop=2 + +/*********************************************************************** + Moses - factored phrase-based language decoder + Copyright (C) 2006 University of Edinburgh + + This library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + This library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with this library; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + ***********************************************************************/ + +#include +#include +#include "EstimatedScores.h" + +using namespace std; + +namespace Moses2 +{ +/** + * Calculate future score estimate for a given coverage bitmap + * + * /param bitmap coverage bitmap + */ + +float EstimatedScores::CalcEstimatedScore(Bitmap const &bitmap) const +{ + const size_t notInGap = numeric_limits::max(); + size_t startGap = notInGap; + float estimatedScore = 0.0f; + for (size_t currPos = 0; currPos < bitmap.GetSize(); currPos++) { + // start of a new gap? + if (bitmap.GetValue(currPos) == false && startGap == notInGap) { + startGap = currPos; + } + // end of a gap? + else if (bitmap.GetValue(currPos) == true && startGap != notInGap) { + estimatedScore += GetValue(startGap, currPos - 1); + startGap = notInGap; + } + } + // coverage ending with gap? + if (startGap != notInGap) { + estimatedScore += GetValue(startGap, bitmap.GetSize() - 1); + } + + return estimatedScore; +} + +/** + * Calculare future score estimate for a given coverage bitmap + * and an additional span that is also covered. This function is used + * to compute future score estimates for hypotheses that we may want + * build, but first want to check. + * + * Note: this function is implemented a bit more complex than + * the basic one (w/o additional phrase) for speed reasons, + * which is probably overkill. + * + * /param bitmap coverage bitmap + * /param startPos start of the span that is added to the coverage + * /param endPos end of the span that is added to the coverage + */ + +float EstimatedScores::CalcEstimatedScore(Bitmap const &bitmap, size_t startPos, + size_t endPos) const +{ + const size_t notInGap = numeric_limits::max(); + float estimatedScore = 0.0f; + size_t startGap = bitmap.GetFirstGapPos(); + if (startGap == NOT_FOUND) return estimatedScore; // everything filled + + // start loop at first gap + size_t startLoop = startGap + 1; + if (startPos == startGap) { // unless covered by phrase + startGap = notInGap; + startLoop = endPos + 1; // -> postpone start + } + + size_t lastCovered = bitmap.GetLastPos(); + if (endPos > lastCovered || lastCovered == NOT_FOUND) lastCovered = endPos; + + for (size_t currPos = startLoop; currPos <= lastCovered; currPos++) { + // start of a new gap? + if (startGap == notInGap && bitmap.GetValue(currPos) == false + && (currPos < startPos || currPos > endPos)) { + startGap = currPos; + } + // end of a gap? + else if (startGap != notInGap + && (bitmap.GetValue(currPos) == true + || (startPos <= currPos && currPos <= endPos))) { + estimatedScore += GetValue(startGap, currPos - 1); + startGap = notInGap; + } + } + // coverage ending with gap? + if (lastCovered != bitmap.GetSize() - 1) { + estimatedScore += GetValue(lastCovered + 1, bitmap.GetSize() - 1); + } + + return estimatedScore; +} + +} + diff --git a/mosesdecoder/moses2/InputPathBase.h b/mosesdecoder/moses2/InputPathBase.h new file mode 100644 index 0000000000000000000000000000000000000000..59fb219e350817df11a0dbe5680f6dbd403d9388 --- /dev/null +++ b/mosesdecoder/moses2/InputPathBase.h @@ -0,0 +1,32 @@ +/* + * InputPath.h + * + * Created on: 23 Oct 2015 + * Author: hieu + */ + +#pragma once + +#include +#include +#include "SubPhrase.h" +#include "legacy/Range.h" + +namespace Moses2 +{ + +class PhraseTable; + +class InputPathBase +{ +public: + const InputPathBase *prefixPath; + Range range; + + InputPathBase(MemPool &pool, const Range &range, + size_t numPt, const InputPathBase *prefixPath); + +}; + +} + diff --git a/mosesdecoder/moses2/InputType.h b/mosesdecoder/moses2/InputType.h new file mode 100644 index 0000000000000000000000000000000000000000..b4f901ac69c16ac8e33e580f9d7966209b8f7aef --- /dev/null +++ b/mosesdecoder/moses2/InputType.h @@ -0,0 +1,86 @@ +/* + * InputType.h + * + * Created on: 14 Dec 2015 + * Author: hieu + */ + +#pragma once + +#include "PhraseBased/ReorderingConstraint.h" +#include "TypeDef.h" + +namespace Moses2 +{ + +class InputType +{ +public: + ////////////////////////////////////////////////////////////////////////////// + class XMLOption + { + public: + size_t startPos, phraseSize; + + SCORE prob; + + XMLOption(MemPool &pool, const std::string &nodeName, size_t vStartPos); + + const char *GetNodeName() const { + return m_nodeName; + } + + const char *GetTranslation() const { + return m_translation; + } + + const char *GetEntity() const { + return m_entity; + } + + void SetTranslation(MemPool &pool, const std::string &val); + void SetEntity(MemPool &pool, const std::string &val); + + std::string Debug(const System &system) const; + public: + char *m_nodeName; + char *m_translation; + char *m_entity; + + }; + + ////////////////////////////////////////////////////////////////////////////// + + InputType(MemPool &pool); + virtual ~InputType(); + + virtual void Init(const System &system, size_t size, int max_distortion); + + ReorderingConstraint &GetReorderingConstraint() { + return m_reorderingConstraint; + } + + const ReorderingConstraint &GetReorderingConstraint() const { + return m_reorderingConstraint; + } + + const Vector &GetXMLOptions() const { + return m_xmlOptions; + } + + void AddXMLOption(const System &system, const XMLOption *xmlOption); + + //! Returns true if there were any XML tags parsed that at least partially covered the range passed + bool XmlOverlap(size_t startPos, size_t endPos) const; + + virtual std::string Debug(const System &system) const; + +protected: + ReorderingConstraint m_reorderingConstraint; /**< limits on reordering specified either by "-mp" switch or xml tags */ + Vector m_xmlOptions; + Vector m_xmlCoverageMap; + +}; + +} /* namespace Moses2 */ + diff --git a/mosesdecoder/moses2/Moses2Wrapper.cpp b/mosesdecoder/moses2/Moses2Wrapper.cpp new file mode 100644 index 0000000000000000000000000000000000000000..a6907af522e889dc27057d83cc6655ccff3f423d --- /dev/null +++ b/mosesdecoder/moses2/Moses2Wrapper.cpp @@ -0,0 +1,70 @@ +#include "Moses2Wrapper.h" +#include "System.h" +#include "legacy/Parameter.h" +#include "TranslationTask.h" +#include + +using namespace std; +namespace Moses2 { + //summary :: need to update the LM path at runtime with complete artifact path. + void Moses2Wrapper::UpdateLMPath(const std::string& filePath) { + + char sep = '/'; + + #ifdef _WIN32 + sep = '\\'; + #endif + auto file = filePath.substr(filePath.find_last_of(sep) + 1); + auto path = filePath.substr(0, filePath.find_last_of(sep)); + auto a = m_param->GetParam("feature"); + std::vector feature; + for (int i = 0; i < a->size(); i++) { + auto abc = Tokenize(a->at(i)); + if (*abc.begin() == "KENLM") { + string s = ""; + for (int k = 0; k < abc.size(); k++) { + if (abc.at(k).find("path=") != string::npos) { + auto lm = abc.at(k).substr(abc.at(k).find_last_of("=") + 1); + s = s + "path=" + path + sep + lm + " "; + } + else { + s = s + abc.at(k) + " "; + } + } + feature.push_back(s.erase(s.find_last_not_of(" \n\r\t") + 1)); + } + else { + feature.push_back(a->at(i)); + } + } + m_param->OverwriteParam("feature", feature); + } + + Moses2Wrapper::Moses2Wrapper(const std::string &filePath) { + m_param = new Parameter(); + m_param->LoadParam(filePath); + UpdateLMPath(filePath); + m_system = new System(*m_param); + } + + std::string Moses2Wrapper::Translate(const std::string &input , long id, bool nbest) { + TranslationTask task(*m_system, input, id); + return task.ReturnTranslation(nbest); + } + Moses2Wrapper::~Moses2Wrapper() { + delete m_param; + delete m_system; + } + + char* Moses2Wrapper::CopyString(const char* str) { + int32_t size = (int32_t)strlen(str); + char* obj = (char*)malloc(size + 1); + memcpy(obj, str, size); + obj[size] = '\0'; + return obj; + } + void Moses2Wrapper::Free(void* ptr) { + free(ptr); + } + +} \ No newline at end of file diff --git a/mosesdecoder/moses2/Phrase.cpp b/mosesdecoder/moses2/Phrase.cpp new file mode 100644 index 0000000000000000000000000000000000000000..dd4abf3283f571321110977df90e6c84a4b8da39 --- /dev/null +++ b/mosesdecoder/moses2/Phrase.cpp @@ -0,0 +1,23 @@ +/* + * PhraseImpl.cpp + * + * Created on: 23 Oct 2015 + * Author: hieu + */ +#include +#include "Phrase.h" +#include "Word.h" +#include "MemPool.h" +#include "Scores.h" +#include "System.h" + +using namespace std; + +namespace Moses2 +{ + + + + +} // namespace + diff --git a/mosesdecoder/moses2/Scores.cpp b/mosesdecoder/moses2/Scores.cpp new file mode 100644 index 0000000000000000000000000000000000000000..6cf12142227574a815acf06b62c8bed9d82ed6b2 --- /dev/null +++ b/mosesdecoder/moses2/Scores.cpp @@ -0,0 +1,283 @@ +/* + * Scores.cpp + * + * Created on: 23 Oct 2015 + * Author: hieu + */ + +#include +#include +#include +#include +#include "Scores.h" +#include "Weights.h" +#include "System.h" +#include "FF/FeatureFunction.h" +#include "FF/FeatureFunctions.h" +#include "legacy/Util2.h" + +using namespace std; + +namespace Moses2 +{ + +Scores::Scores(const System &system, MemPool &pool, size_t numScores) : + m_total(0) +{ + if (system.options.nbest.nbest_size) { + m_scores = new (pool.Allocate(numScores)) SCORE[numScores]; + Init(m_scores, numScores, 0); + } else { + m_scores = NULL; + } +} + +Scores::Scores(const System &system, MemPool &pool, size_t numScores, + const Scores &origScores) : + m_total(origScores.m_total) +{ + if (system.options.nbest.nbest_size) { + m_scores = new (pool.Allocate(numScores)) SCORE[numScores]; + memcpy(m_scores, origScores.m_scores, sizeof(SCORE) * numScores); + } else { + m_scores = NULL; + } +} + +Scores::~Scores() +{ + +} + +const SCORE *Scores::GetScores(const FeatureFunction &featureFunction) const +{ + assert(m_scores); + size_t ffStartInd = featureFunction.GetStartInd(); + const SCORE &scores = m_scores[ffStartInd]; + return &scores; +} + +void Scores::Reset(const System &system) +{ + if (system.options.nbest.nbest_size) { + size_t numScores = system.featureFunctions.GetNumScores(); + Init(m_scores, numScores, 0); + } + m_total = 0; +} + +void Scores::PlusEquals(const System &system, + const FeatureFunction &featureFunction, const SCORE &score) +{ + assert(featureFunction.GetNumScores() == 1); + + const Weights &weights = system.weights; + + size_t ffStartInd = featureFunction.GetStartInd(); + if (system.options.nbest.nbest_size) { + m_scores[ffStartInd] += score; + } + SCORE weight = weights[ffStartInd]; + m_total += score * weight; +} + +void Scores::PlusEquals(const System &system, + const FeatureFunction &featureFunction, const SCORE &score, size_t offset) +{ + assert(offset < featureFunction.GetNumScores()); + + const Weights &weights = system.weights; + + size_t ffStartInd = featureFunction.GetStartInd(); + if (system.options.nbest.nbest_size) { + m_scores[ffStartInd + offset] += score; + } + SCORE weight = weights[ffStartInd + offset]; + m_total += score * weight; +} + +void Scores::PlusEquals(const System &system, + const FeatureFunction &featureFunction, const std::vector &scores) +{ + assert(scores.size() == featureFunction.GetNumScores()); + + const Weights &weights = system.weights; + + size_t ffStartInd = featureFunction.GetStartInd(); + for (size_t i = 0; i < scores.size(); ++i) { + SCORE incrScore = scores[i]; + if (system.options.nbest.nbest_size) { + m_scores[ffStartInd + i] += incrScore; + } + //cerr << "ffStartInd=" << ffStartInd << " " << i << endl; + SCORE weight = weights[ffStartInd + i]; + m_total += incrScore * weight; + } +} + +void Scores::PlusEquals(const System &system, + const FeatureFunction &featureFunction, SCORE scores[]) +{ + //assert(scores.size() == featureFunction.GetNumScores()); + + const Weights &weights = system.weights; + + size_t ffStartInd = featureFunction.GetStartInd(); + for (size_t i = 0; i < featureFunction.GetNumScores(); ++i) { + SCORE incrScore = scores[i]; + if (system.options.nbest.nbest_size) { + m_scores[ffStartInd + i] += incrScore; + } + //cerr << "ffStartInd=" << ffStartInd << " " << i << endl; + SCORE weight = weights[ffStartInd + i]; + m_total += incrScore * weight; + } +} + +void Scores::PlusEquals(const System &system, const Scores &other) +{ + size_t numScores = system.featureFunctions.GetNumScores(); + if (system.options.nbest.nbest_size) { + for (size_t i = 0; i < numScores; ++i) { + m_scores[i] += other.m_scores[i]; + } + } + m_total += other.m_total; +} + +void Scores::MinusEquals(const System &system, const Scores &other) +{ + size_t numScores = system.featureFunctions.GetNumScores(); + if (system.options.nbest.nbest_size) { + for (size_t i = 0; i < numScores; ++i) { + m_scores[i] -= other.m_scores[i]; + } + } + m_total -= other.m_total; +} + +void Scores::Assign(const System &system, + const FeatureFunction &featureFunction, const SCORE &score) +{ + assert(featureFunction.GetNumScores() == 1); + + const Weights &weights = system.weights; + + size_t ffStartInd = featureFunction.GetStartInd(); + + if (system.options.nbest.nbest_size) { + assert(m_scores[ffStartInd] == 0); + m_scores[ffStartInd] = score; + } + SCORE weight = weights[ffStartInd]; + m_total += score * weight; + +} + +void Scores::Assign(const System &system, + const FeatureFunction &featureFunction, const std::vector &scores) +{ + assert(scores.size() == featureFunction.GetNumScores()); + + const Weights &weights = system.weights; + + size_t ffStartInd = featureFunction.GetStartInd(); + for (size_t i = 0; i < scores.size(); ++i) { + SCORE incrScore = scores[i]; + + if (system.options.nbest.nbest_size) { + assert(m_scores[ffStartInd + i] == 0); + m_scores[ffStartInd + i] = incrScore; + } + //cerr << "ffStartInd=" << ffStartInd << " " << i << endl; + SCORE weight = weights[ffStartInd + i]; + m_total += incrScore * weight; + } +} + +void Scores::CreateFromString(const std::string &str, + const FeatureFunction &featureFunction, const System &system, + bool transformScores) +{ + vector scores = Tokenize(str); + if (transformScores) { + std::transform(scores.begin(), scores.end(), scores.begin(), + TransformScore); + std::transform(scores.begin(), scores.end(), scores.begin(), FloorScore); + } + + /* + std::copy(scores.begin(),scores.end(), + std::ostream_iterator(cerr," ")); + */ + + PlusEquals(system, featureFunction, scores); +} + +std::string Scores::Debug(const System &system) const +{ + stringstream out; + out << "total=" << m_total; + + if (system.options.nbest.nbest_size) { + out << ", "; + BOOST_FOREACH(const FeatureFunction *ff, system.featureFunctions.GetFeatureFunctions()) { + out << ff->GetName() << "= "; + for (size_t i = ff->GetStartInd(); i < (ff->GetStartInd() + ff->GetNumScores()); ++i) { + out << m_scores[i] << " "; + } + } + } + + return out.str(); +} + +void Scores::OutputBreakdown(std::ostream &out, const System &system) const +{ + if (system.options.nbest.nbest_size) { + BOOST_FOREACH(const FeatureFunction *ff, system.featureFunctions.GetFeatureFunctions()) { + if (ff->IsTuneable()) { + out << ff->GetName() << "= "; + for (size_t i = ff->GetStartInd(); i < (ff->GetStartInd() + ff->GetNumScores()); ++i) { + out << m_scores[i] << " "; + } + } + } + } +} + +// static functions to work out estimated scores +SCORE Scores::CalcWeightedScore(const System &system, + const FeatureFunction &featureFunction, SCORE scores[]) +{ + SCORE ret = 0; + + const Weights &weights = system.weights; + + size_t ffStartInd = featureFunction.GetStartInd(); + for (size_t i = 0; i < featureFunction.GetNumScores(); ++i) { + SCORE incrScore = scores[i]; + + //cerr << "ffStartInd=" << ffStartInd << " " << i << endl; + SCORE weight = weights[ffStartInd + i]; + ret += incrScore * weight; + } + + return ret; +} + +SCORE Scores::CalcWeightedScore(const System &system, + const FeatureFunction &featureFunction, SCORE score) +{ + const Weights &weights = system.weights; + assert(featureFunction.GetNumScores() == 1); + + size_t ffStartInd = featureFunction.GetStartInd(); + SCORE weight = weights[ffStartInd]; + SCORE ret = score * weight; + + return ret; +} + +} + diff --git a/mosesdecoder/moses2/System.cpp b/mosesdecoder/moses2/System.cpp new file mode 100644 index 0000000000000000000000000000000000000000..b46690d1a75cb595f3fdbaafefd631d3273256e2 --- /dev/null +++ b/mosesdecoder/moses2/System.cpp @@ -0,0 +1,262 @@ +/* + * System.cpp + * + * Created on: 23 Oct 2015 + * Author: hieu + */ +#include +#include +#include +#include +#include +#include "System.h" +#include "FF/FeatureFunction.h" +#include "TranslationModel/UnknownWordPenalty.h" +#include "legacy/Util2.h" +#include "util/exception.hh" + +using namespace std; + +namespace Moses2 +{ +#ifndef WIN32 + thread_local MemPool System::m_managerPool; + thread_local MemPool System::m_systemPool; + thread_local Recycler System::m_hypoRecycler; +#endif // WIN32 + +System::System(const Parameter ¶msArg) : + params(paramsArg), featureFunctions(*this) +{ + options.init(paramsArg); + IsPb(); + + bestCollector.reset(new OutputCollector()); + + params.SetParameter(cpuAffinityOffset, "cpu-affinity-offset", -1); + params.SetParameter(cpuAffinityOffsetIncr, "cpu-affinity-increment", 1); + + const PARAM_VEC *section; + + // output collectors + if (options.nbest.nbest_size && options.nbest.output_file_path != "-") { + nbestCollector.reset(new OutputCollector(options.nbest.output_file_path)); + } + + if (!options.output.detailed_transrep_filepath.empty()) { + detailedTranslationCollector.reset(new OutputCollector(options.output.detailed_transrep_filepath)); + } + + featureFunctions.Create(); + LoadWeights(); + + if (params.GetParam("show-weights")) { + cerr << "Showing weights then exit" << endl; + featureFunctions.ShowWeights(weights); + //return; + } + + cerr << "START featureFunctions.Load()" << endl; + featureFunctions.Load(); + cerr << "START LoadMappings()" << endl; + LoadMappings(); + cerr << "END LoadMappings()" << endl; + LoadDecodeGraphBackoff(); + cerr << "END LoadDecodeGraphBackoff()" << endl; + + UTIL_THROW_IF2(options.input.xml_policy == XmlConstraint, "XmlConstraint not supported"); + + // max spans for scfg decoding + if (!isPb) { + section = params.GetParam("max-chart-span"); + if (section && section->size()) { + maxChartSpans = Scan(*section); + maxChartSpans.resize(mappings.size(), DEFAULT_MAX_CHART_SPAN); + + /* + cerr << "maxChartSpans=" << maxChartSpans.size(); + for (size_t i = 0; i < maxChartSpans.size(); ++i) { + cerr << " " << mappings[i]->GetName() << "=" << maxChartSpans[i]; + } + cerr << endl; + */ + } + } + +} + +System::~System() +{ +} + +void System::LoadWeights() +{ + weights.Init(featureFunctions); + + //cerr << "Weights:" << endl; + typedef std::map > WeightMap; + const WeightMap &allWeights = params.GetAllWeights(); + + // check all weights are there for all FF + const std::vector &ffs = featureFunctions.GetFeatureFunctions(); + BOOST_FOREACH(const FeatureFunction *ff, ffs) { + if (ff->IsTuneable()) { + const std::string &ffName = ff->GetName(); + WeightMap::const_iterator iterWeight = allWeights.find(ffName); + UTIL_THROW_IF2(iterWeight == allWeights.end(), "Must specify weight for " << ffName); + } + } + + + // set weight + BOOST_FOREACH(const WeightMap::value_type &valPair, allWeights) { + const string &ffName = valPair.first; + const std::vector &ffWeights = valPair.second; + /* + cerr << ffName << "="; + for (size_t i = 0; i < ffWeights.size(); ++i) { + cerr << ffWeights[i] << " "; + } + cerr << endl; + */ + weights.SetWeights(featureFunctions, ffName, ffWeights); + } +} + +void System::LoadMappings() +{ + const PARAM_VEC *vec = params.GetParam("mapping"); + UTIL_THROW_IF2(vec == NULL, "Must have [mapping] section"); + + BOOST_FOREACH(const std::string &line, *vec) { + vector toks = Tokenize(line); + assert( (toks.size() == 2 && toks[0] == "T") || (toks.size() == 3 && toks[1] == "T") ); + + size_t ptInd; + if (toks.size() == 2) { + ptInd = Scan(toks[1]); + } else { + ptInd = Scan(toks[2]); + } + const PhraseTable *pt = featureFunctions.GetPhraseTableExcludeUnknownWordPenalty(ptInd); + mappings.push_back(pt); + } + +// unk pt + const UnknownWordPenalty *unkWP = featureFunctions.GetUnknownWordPenalty(); + if (unkWP) { + mappings.push_back(unkWP); + } +} + +void System::LoadDecodeGraphBackoff() +{ + const PARAM_VEC *vec = params.GetParam("decoding-graph-backoff"); + + for (size_t i = 0; i < mappings.size(); ++i) { + PhraseTable *pt = const_cast(mappings[i]); + + if (vec && vec->size() < i) { + pt->decodeGraphBackoff = Scan((*vec)[i]); + } else if (pt == featureFunctions.GetUnknownWordPenalty()) { + pt->decodeGraphBackoff = 1; + } else { + pt->decodeGraphBackoff = 0; + } + } +} + +void System::IsPb() +{ + switch (options.search.algo) { + case Normal: + case NormalBatch: + case CubePruning: + case CubePruningPerMiniStack: + case CubePruningPerBitmap: + case CubePruningCardinalStack: + case CubePruningBitmapStack: + case CubePruningMiniStack: + isPb = true; + break; + case CYKPlus: + isPb = false; + break; + default: + throw std::runtime_error("Unknown search algorithm " + options.search.algo); + break; + } +} + +FactorCollection& System::GetVocab() const +{ + return m_vocab; +} + +////////////////////////////////////////////////////// +// thread local stuff +Batch& System::GetBatch(MemPool& pool) const +{ + Batch* obj; + obj = m_batch.get(); + if (obj == NULL) { + obj = new Batch(pool); + m_batch.reset(obj); + } + assert(obj); + return *obj; +} + +#ifdef WIN32 +template +C& GetThreadSpecificObj(boost::thread_specific_ptr &threadSpecificPtr) +{ + C* obj; + obj = threadSpecificPtr.get(); + if (obj == NULL) { + obj = new C(); + threadSpecificPtr.reset(obj); + } + assert(obj); + return *obj; +} + +MemPool& System::GetManagerPool() const +{ + MemPool &obj = GetThreadSpecificObj(m_managerPool); + return obj; +} + +MemPool& System::GetSystemPool() const +{ + MemPool& obj = GetThreadSpecificObj(m_systemPool); + return obj; +} + +Recycler& System::GetHypoRecycler() const +{ + Recycler& obj = GetThreadSpecificObj >(m_hypoRecycler); + return obj; +} + +#else +MemPool& System::GetManagerPool() const +{ + return m_managerPool; +} + +MemPool& System::GetSystemPool() const +{ + return m_systemPool; +} + +Recycler& System::GetHypoRecycler() const +{ + return m_hypoRecycler; +} + +#endif + + +} + diff --git a/mosesdecoder/moses2/System.h b/mosesdecoder/moses2/System.h new file mode 100644 index 0000000000000000000000000000000000000000..009d2647e326be44dbf0d927632be8a9e21c1488 --- /dev/null +++ b/mosesdecoder/moses2/System.h @@ -0,0 +1,90 @@ +/* + * System.h + * + * Created on: 23 Oct 2015 + * Author: hieu + */ + +#pragma once +#include +#include +#include +#include +#include +#include "FF/FeatureFunctions.h" +#include "Weights.h" +#include "MemPool.h" +#include "Recycler.h" +#include "legacy/FactorCollection.h" +#include "legacy/Parameter.h" +#include "TypeDef.h" +#include "legacy/Bitmaps.h" +#include "legacy/OutputCollector.h" +#include "parameters/AllOptions.h" + +namespace Moses2 +{ +namespace NSCubePruning +{ +class Stack; +} + +class FeatureFunction; +class StatefulFeatureFunction; +class PhraseTable; +class HypothesisBase; + +class System +{ +public: + const Parameter ¶ms; + AllOptions options; + FeatureFunctions featureFunctions; + Weights weights; + std::vector mappings; + + std::vector maxChartSpans; + bool isPb; + + mutable boost::shared_ptr bestCollector, nbestCollector, detailedTranslationCollector; + + // moses.ini params + int cpuAffinityOffset; + int cpuAffinityOffsetIncr; + + System(const Parameter ¶msArg); + virtual ~System(); + + MemPool &GetSystemPool() const; + MemPool &GetManagerPool() const; + FactorCollection &GetVocab() const; + + Recycler &GetHypoRecycler() const; + + Batch &GetBatch(MemPool &pool) const; + +protected: + mutable FactorCollection m_vocab; + + mutable boost::thread_specific_ptr m_batch; + +#ifdef WIN32 + mutable boost::thread_specific_ptr m_managerPool; + mutable boost::thread_specific_ptr m_systemPool; + mutable boost::thread_specific_ptr > m_hypoRecycler; +#else + thread_local static MemPool m_managerPool; + thread_local static MemPool m_systemPool; + thread_local static Recycler m_hypoRecycler; +#endif + + void LoadWeights(); + void LoadMappings(); + void LoadDecodeGraphBackoff(); + + void IsPb(); + +}; + +} + diff --git a/mosesdecoder/moses2/TargetPhrase.h b/mosesdecoder/moses2/TargetPhrase.h new file mode 100644 index 0000000000000000000000000000000000000000..2522f85df5d2c9eb8763de84591a7bfdf475cef1 --- /dev/null +++ b/mosesdecoder/moses2/TargetPhrase.h @@ -0,0 +1,164 @@ +/* + * TargetPhrase.h + * + * Created on: 26 Apr 2016 + * Author: hieu + */ + +#pragma once +#include +#include "PhraseImplTemplate.h" +#include "System.h" +#include "Scores.h" +#include "AlignmentInfoCollection.h" +#include "TranslationModel/PhraseTable.h" + +namespace Moses2 +{ +class AlignmentInfo; + +template +class TargetPhrase: public PhraseImplTemplate +{ +public: + typedef PhraseImplTemplate Parent; + const PhraseTable &pt; + mutable void **ffData; + SCORE *scoreProperties; + + TargetPhrase(MemPool &pool, const PhraseTable &pt, const System &system, size_t size) + : PhraseImplTemplate(pool, size) + , pt(pt) + , scoreProperties(NULL) + , m_alignTerm(&AlignmentInfoCollection::Instance().GetEmptyAlignmentInfo()) { + m_scores = new (pool.Allocate()) Scores(system, pool, + system.featureFunctions.GetNumScores()); + } + + Scores &GetScores() { + return *m_scores; + } + + const Scores &GetScores() const { + return *m_scores; + } + + virtual SCORE GetScoreForPruning() const = 0; + + SCORE *GetScoresProperty(int propertyInd) const { + return scoreProperties ? scoreProperties + propertyInd : NULL; + } + + const AlignmentInfo &GetAlignTerm() const { + return *m_alignTerm; + } + + void SetAlignTerm(const AlignmentInfo &alignInfo) { + m_alignTerm = &alignInfo; + } + + // ALNREP = alignment representation, + // see AlignmentInfo constructors for supported representations + template + void + SetAlignTerm(const ALNREP &coll) { + m_alignTerm = AlignmentInfoCollection::Instance().Add(coll); + } + + virtual void SetAlignmentInfo(const std::string &alignString) { + AlignmentInfo::CollType alignTerm; + + std::vector toks = Tokenize(alignString); + for (size_t i = 0; i < toks.size(); ++i) { + std::vector alignPair = Tokenize(toks[i], "-"); + UTIL_THROW_IF2(alignPair.size() != 2, "Wrong alignment format"); + + size_t sourcePos = alignPair[0]; + size_t targetPos = alignPair[1]; + + alignTerm.insert(std::pair(sourcePos, targetPos)); + } + + SetAlignTerm(alignTerm); + // cerr << "TargetPhrase::SetAlignmentInfo(const StringPiece &alignString) this:|" << *this << "|\n"; + + //cerr << "alignTerm=" << alignTerm.size() << endl; + //cerr << "alignNonTerm=" << alignNonTerm.size() << endl; + + } + + void OutputToStream(const System &system, const Phrase &inputPhrase, std::ostream &out) const { + // get placeholders + FactorType placeholderFactor = system.options.input.placeholder_factor; + std::map placeholders; + if (placeholderFactor != NOT_FOUND) { + // creates map of target position -> factor for placeholders + placeholders = GetPlaceholders(system, inputPhrase); + } + + size_t size = PhraseImplTemplate::GetSize(); + for (size_t i = 0; i < size; ++i) { + // output placeholder, if any + std::map::const_iterator iter = placeholders.find(i); + if (iter == placeholders.end()) { + const WORD &word = (*this)[i]; + word.OutputToStream(system, out); + } else { + const Factor *factor = iter->second; + out << *factor; + } + + out << " "; + } + } + + std::map GetPlaceholders(const System &system, const Phrase &inputPhrase) const { + FactorType placeholderFactor = system.options.input.placeholder_factor; + std::map ret; + //std::cerr << "inputPhrase=" << inputPhrase.Debug(system) << std::endl; + + for (size_t sourcePos = 0; sourcePos < inputPhrase.GetSize(); ++sourcePos) { + const Factor *factor = inputPhrase[sourcePos][placeholderFactor]; + if (factor) { + //std::cerr << "factor=" << *factor << std::endl; + //std::cerr << "tp=" << Debug(system) << std::endl; + std::set targetPos = GetAlignTerm().GetAlignmentsForSource(sourcePos); + UTIL_THROW_IF2(targetPos.size() != 1, + "Placeholder should be aligned to 1, and only 1, word:" << targetPos.size() << "!=1"); + ret[*targetPos.begin()] = factor; + } + } + + return ret; + } + + virtual std::string Debug(const System &system) const { + std::stringstream out; + out << Phrase::Debug(system); + out << " pt=" << pt.GetName() << " "; + out << " SCORES:" << GetScores().Debug(system); + out << " ALIGN-T:"; + out << GetAlignTerm().Debug(system); + + return out.str(); + } + +protected: + Scores *m_scores; + const AlignmentInfo *m_alignTerm; +}; + +/////////////////////////////////////////////////////////////////////// +template +struct CompareScoreForPruning { + bool operator()(const TP *a, const TP *b) const { + return a->GetScoreForPruning() > b->GetScoreForPruning(); + } + + bool operator()(const TP &a, const TP &b) const { + return a.GetScoreForPruning() > b.GetScoreForPruning(); + } +}; + +} /* namespace Moses2a */ + diff --git a/mosesdecoder/moses2/TrellisPaths.cpp b/mosesdecoder/moses2/TrellisPaths.cpp new file mode 100644 index 0000000000000000000000000000000000000000..814da45211235ad584c87883bf6779855abb38fc --- /dev/null +++ b/mosesdecoder/moses2/TrellisPaths.cpp @@ -0,0 +1,14 @@ +/* + * TrellisPaths.cpp + * + * Created on: 16 Mar 2016 + * Author: hieu + */ +#include "TrellisPaths.h" +#include "legacy/Util2.h" + +namespace Moses2 +{ + + +} /* namespace Moses2 */ diff --git a/mosesdecoder/moses2/TypeDef.cpp b/mosesdecoder/moses2/TypeDef.cpp new file mode 100644 index 0000000000000000000000000000000000000000..b8b79c59c642eed8413b8f2a177216287cee3f16 --- /dev/null +++ b/mosesdecoder/moses2/TypeDef.cpp @@ -0,0 +1,11 @@ +#include "TypeDef.h" +#include "util/exception.hh" +#include + + +namespace Moses2 +{ + + + +} diff --git a/mosesdecoder/moses2/Word.h b/mosesdecoder/moses2/Word.h new file mode 100644 index 0000000000000000000000000000000000000000..9d742eece61ed26bf7b4ff64dae481175be7d069 --- /dev/null +++ b/mosesdecoder/moses2/Word.h @@ -0,0 +1,63 @@ +/* + * Word.h + * + * Created on: 23 Oct 2015 + * Author: hieu + */ + +#pragma once + +#include +#include "TypeDef.h" +#include "legacy/Factor.h" +#include "legacy/FactorCollection.h" + +namespace Moses2 +{ + +class Word +{ +public: + explicit Word(); + Word(const Word ©); + + virtual ~Word(); + + void CreateFromString(FactorCollection &vocab, const System &system, + const std::string &str); + + virtual size_t hash() const; + virtual size_t hash(const std::vector &factors) const; + + int Compare(const Word &compare) const; + + virtual bool operator==(const Word &compare) const { + int cmp = Compare(compare); + return cmp == 0; + } + + virtual bool operator!=(const Word &compare) const { + return !((*this) == compare); + } + + virtual bool operator<(const Word &compare) const; + + const Factor* operator[](size_t ind) const { + return m_factors[ind]; + } + + const Factor*& operator[](size_t ind) { + return m_factors[ind]; + } + + virtual void OutputToStream(const System &system, std::ostream &out) const; + virtual std::string Debug(const System &system) const; + + std::string GetString(const FactorList &factorTypes) const; +protected: + const Factor *m_factors[MAX_NUM_FACTORS]; + +}; + +} + diff --git a/mosesdecoder/phrase-extract/extract-ghkm/AlignmentGraph.cpp b/mosesdecoder/phrase-extract/extract-ghkm/AlignmentGraph.cpp new file mode 100644 index 0000000000000000000000000000000000000000..21708bdfcb942c77219083d6025b83ca71b5e7df --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-ghkm/AlignmentGraph.cpp @@ -0,0 +1,418 @@ +/*********************************************************************** + Moses - statistical machine translation system + Copyright (C) 2006-2011 University of Edinburgh + + This library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + This library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with this library; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +***********************************************************************/ + +#include "AlignmentGraph.h" + +#include +#include +#include +#include +#include + +#include "SyntaxTree.h" + +#include "ComposedRule.h" +#include "Node.h" +#include "Options.h" +#include "Subgraph.h" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace GHKM +{ + +AlignmentGraph::AlignmentGraph(const SyntaxTree *t, + const std::vector &s, + const Alignment &a) +{ + // Copy the parse tree nodes and add them to m_targetNodes. + m_root = CopyParseTree(t); + + // Create a node for each source word. + m_sourceNodes.reserve(s.size()); + for (std::vector::const_iterator p(s.begin()); + p != s.end(); ++p) { + m_sourceNodes.push_back(new Node(*p, SOURCE)); + } + + // Connect source nodes to parse tree leaves according to the given word + // alignment. + std::vector targetTreeLeaves; + GetTargetTreeLeaves(m_root, targetTreeLeaves); + for (Alignment::const_iterator p(a.begin()); p != a.end(); ++p) { + Node *src = m_sourceNodes[p->first]; + Node *tgt = targetTreeLeaves[p->second]; + src->AddParent(tgt); + tgt->AddChild(src); + } + + // Attach unaligned source words (if any). + AttachUnalignedSourceWords(); + + // Populate node spans. + std::vector::const_iterator p(m_sourceNodes.begin()); + for (int i = 0; p != m_sourceNodes.end(); ++p, ++i) { + (*p)->PropagateIndex(i); + } + + // Calculate complement spans. + CalcComplementSpans(m_root); +} + +AlignmentGraph::~AlignmentGraph() +{ + for (std::vector::iterator p(m_sourceNodes.begin()); + p != m_sourceNodes.end(); ++p) { + delete *p; + } + for (std::vector::iterator p(m_targetNodes.begin()); + p != m_targetNodes.end(); ++p) { + delete *p; + } +} + +Subgraph AlignmentGraph::ComputeMinimalFrontierGraphFragment( + Node *root, + const std::set &frontierSet) +{ + std::stack expandableNodes; + std::set expandedNodes; + + if (root->IsSink()) { + expandedNodes.insert(root); + } else { + expandableNodes.push(root); + } + + while (!expandableNodes.empty()) { + Node *n = expandableNodes.top(); + expandableNodes.pop(); + + const std::vector &children = n->GetChildren(); + + for (std::vector::const_iterator p(children.begin()); + p != children.end(); ++p) { + Node *child = *p; + if (child->IsSink()) { + expandedNodes.insert(child); + continue; + } + std::set::const_iterator q = frontierSet.find(child); + if (q == frontierSet.end()) { //child is not from the frontier set + expandableNodes.push(child); + } else if (child->GetType() == TARGET) { // still need source word + expandableNodes.push(child); + } else { + expandedNodes.insert(child); + } + } + } + + return Subgraph(root, expandedNodes); +} + +void AlignmentGraph::ExtractMinimalRules(const Options &options) +{ + // Determine which nodes are frontier nodes. + std::set frontierSet; + ComputeFrontierSet(m_root, options, frontierSet); + + // Form the minimal frontier graph fragment rooted at each frontier node. + std::vector fragments; + fragments.reserve(frontierSet.size()); + for (std::set::iterator p(frontierSet.begin()); + p != frontierSet.end(); ++p) { + Node *root = *p; + Subgraph fragment = ComputeMinimalFrontierGraphFragment(root, frontierSet); + assert(!fragment.IsTrivial()); + // Can it form an SCFG rule? + // FIXME Does this exclude non-lexical unary rules? + if (root->GetType() == TREE && !root->GetSpan().empty()) { + root->AddRule(new Subgraph(fragment)); + } + } +} + +void AlignmentGraph::ExtractComposedRules(const Options &options) +{ + ExtractComposedRules(m_root, options); +} + +void AlignmentGraph::ExtractComposedRules(Node *node, const Options &options) +{ + // Extract composed rules for all children first. + const std::vector &children = node->GetChildren(); + for (std::vector::const_iterator p(children.begin()); + p != children.end(); ++p) { + ExtractComposedRules(*p, options); + } + + // If there is no minimal rule for this node then there are no composed + // rules. + const std::vector &rules = node->GetRules(); + assert(rules.size() <= 1); + if (rules.empty()) { + return; + } + + // Construct an initial composition candidate from the minimal rule. + ComposedRule cr(*(rules[0])); + if (!cr.GetOpenAttachmentPoint()) { + // No composition possible. + return; + } + + std::queue queue; + queue.push(cr); + while (!queue.empty()) { + ComposedRule cr = queue.front(); + queue.pop(); + const Node *attachmentPoint = cr.GetOpenAttachmentPoint(); + assert(attachmentPoint); + assert(attachmentPoint != node); + // Create all possible rules by composing this node's minimal rule with the + // existing rules (both minimal and composed) rooted at the first open + // attachment point. + const std::vector &rules = attachmentPoint->GetRules(); + for (std::vector::const_iterator p = rules.begin(); + p != rules.end(); ++p) { + assert((*p)->GetRoot()->GetType() == TREE); + ComposedRule *cr2 = cr.AttemptComposition(**p, options); + if (cr2) { + node->AddRule(new Subgraph(cr2->CreateSubgraph())); + if (cr2->GetOpenAttachmentPoint()) { + queue.push(*cr2); + } + delete cr2; + } + } + // Done with this attachment point. Advance to the next, if any. + cr.CloseAttachmentPoint(); + if (cr.GetOpenAttachmentPoint()) { + queue.push(cr); + } + } +} + +Node *AlignmentGraph::CopyParseTree(const SyntaxTree *root) +{ + NodeType nodeType = (root->IsLeaf()) ? TARGET : TREE; + + std::auto_ptr n(new Node(root->value().label, nodeType)); + + if (nodeType == TREE) { + float score = 0.0f; + SyntaxNode::AttributeMap::const_iterator p = + root->value().attributes.find("pcfg"); + if (p != root->value().attributes.end()) { + score = std::atof(p->second.c_str()); + } + n->SetPcfgScore(score); + } + + const std::vector &children = root->children(); + std::vector childNodes; + childNodes.reserve(children.size()); + for (std::vector::const_iterator p(children.begin()); + p != children.end(); ++p) { + Node *child = CopyParseTree(*p); + child->AddParent(n.get()); + childNodes.push_back(child); + } + n->SetChildren(childNodes); + + Node *p = n.release(); + m_targetNodes.push_back(p); + return p; +} + +// Recursively constructs the set of frontier nodes for the tree (or subtree) +// rooted at the given node. +void AlignmentGraph::ComputeFrontierSet(Node *root, + const Options &options, + std::set &frontierSet) const +{ + // Non-tree nodes and unaligned target subtrees are not frontier nodes (and + // nor are their descendants). See the comment for the function + // AlignmentGraph::IsFrontierNode(). + if (root->GetType() != TREE || root->GetSpan().empty()) { + return; + } + + if (IsFrontierNode(*root, options)) { + frontierSet.insert(root); + } + + // Recursively check descendants. + const std::vector &children = root->GetChildren(); + for (std::vector::const_iterator p(children.begin()); + p != children.end(); ++p) { + ComputeFrontierSet(*p, options, frontierSet); + } +} + +// Determines whether the given node is a frontier node or not. The definition +// of a frontier node differs from Galley et al's (2004) in the following ways: +// +// 1. A node with an empty span is not a frontier node (this is to exclude +// unaligned target subtrees). +// 2. Target word nodes are not frontier nodes. +// 3. Source word nodes are not frontier nodes. +// 4. Unless the --AllowUnary option is used, a node is not a frontier node if +// it has the same span as its parent. +bool AlignmentGraph::IsFrontierNode(const Node &n, const Options &options) const +{ + // Don't include word nodes or unaligned target subtrees. + if (n.GetType() != TREE || n.GetSpan().empty()) { + return false; + } + // This is the original GHKM definition of a frontier node. + if (SpansIntersect(n.GetComplementSpan(), Closure(n.GetSpan()))) { + return false; + } + // Unless unary rules are explicitly allowed, we use Chung et al's (2011) + // modified defintion of a frontier node to eliminate the production of + // non-lexical unary rules. + assert(n.GetParents().size() <= 1); + if (!options.allowUnary && + !n.GetParents().empty() && + n.GetParents()[0]->GetSpan() == n.GetSpan()) { + return false; + } + return true; +} + +void AlignmentGraph::CalcComplementSpans(Node *root) +{ + Span compSpan; + std::set siblings; + + const std::vector &parents = root->GetParents(); + for (std::vector::const_iterator p(parents.begin()); + p != parents.end(); ++p) { + const Span &parentCompSpan = (*p)->GetComplementSpan(); + compSpan.insert(parentCompSpan.begin(), parentCompSpan.end()); + const std::vector &c = (*p)->GetChildren(); + siblings.insert(c.begin(), c.end()); + } + + for (std::set::iterator p(siblings.begin()); + p != siblings.end(); ++p) { + if (*p == root) { + continue; + } + const Span &siblingSpan = (*p)->GetSpan(); + compSpan.insert(siblingSpan.begin(), siblingSpan.end()); + } + + root->SetComplementSpan(compSpan); + + const std::vector &children = root->GetChildren(); + for (std::vector::const_iterator p(children.begin()); + p != children.end(); ++p) { + CalcComplementSpans(*p); + } +} + +void AlignmentGraph::GetTargetTreeLeaves(Node *root, + std::vector &leaves) +{ + if (root->IsSink()) { + leaves.push_back(root); + } else { + const std::vector &children = root->GetChildren(); + for (std::vector::const_iterator p(children.begin()); + p != children.end(); ++p) { + GetTargetTreeLeaves(*p, leaves); + } + } +} + +void AlignmentGraph::AttachUnalignedSourceWords() +{ + // Find the unaligned source words (if any). + std::set unaligned; + for (size_t i = 0; i < m_sourceNodes.size(); ++i) { + const Node &sourceNode = (*m_sourceNodes[i]); + if (sourceNode.GetParents().empty()) { + unaligned.insert(i); + } + } + + // Determine the attachment point for each one and attach it. + for (std::set::iterator p = unaligned.begin(); + p != unaligned.end(); ++p) { + int index = *p; + Node *attachmentPoint = DetermineAttachmentPoint(index); + Node *sourceNode = m_sourceNodes[index]; + attachmentPoint->AddChild(sourceNode); + sourceNode->AddParent(attachmentPoint); + } +} + +Node *AlignmentGraph::DetermineAttachmentPoint(int index) +{ + // Find the nearest aligned neighbour to the left, if any. + int i = index; + while (--i >= 0) { + if (!m_sourceNodes[i]->GetParents().empty()) { + break; + } + } + // No aligned neighbours to the left, so attach to the root. + if (i == -1) { + return m_root; + } + // Find the nearest aligned neighbour to the right, if any. + size_t j = index; + while (++j < m_sourceNodes.size()) { + if (!m_sourceNodes[j]->GetParents().empty()) { + break; + } + } + // No aligned neighbours to the right, so attach to the root. + if (j == m_sourceNodes.size()) { + return m_root; + } + // Construct the set of target nodes that are aligned to the left and right + // neighbours. + const std::vector &leftParents = m_sourceNodes[i]->GetParents(); + assert(!leftParents.empty()); + const std::vector &rightParents = m_sourceNodes[j]->GetParents(); + assert(!rightParents.empty()); + std::set targetSet; + targetSet.insert(leftParents.begin(), leftParents.end()); + targetSet.insert(rightParents.begin(), rightParents.end()); + // The attachment point is the lowest common ancestor of the target word + // nodes, unless the LCA is itself a target word, in which case the LCA + // is the parent. This is to avoid including introducing new word alignments. + // It assumes that the parse tree uses preterminals for parts of speech. + Node *lca = Node::LowestCommonAncestor(targetSet.begin(), targetSet.end()); + if (lca->GetType() == TARGET) { + assert(lca->GetParents().size() == 1); + return lca->GetParents()[0]; + } + return lca; +} + +} // namespace GHKM +} // namespace Syntax +} // namespace MosesTraining diff --git a/mosesdecoder/phrase-extract/extract-ghkm/AlignmentGraph.h b/mosesdecoder/phrase-extract/extract-ghkm/AlignmentGraph.h new file mode 100644 index 0000000000000000000000000000000000000000..be1182c1672b5d59a35c8fb5ecbba8ed5ead0758 --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-ghkm/AlignmentGraph.h @@ -0,0 +1,87 @@ +/*********************************************************************** + Moses - statistical machine translation system + Copyright (C) 2006-2011 University of Edinburgh + + This library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + This library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with this library; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +***********************************************************************/ + +#pragma once +#ifndef EXTRACT_GHKM_ALIGNMENT_GRAPH_H_ +#define EXTRACT_GHKM_ALIGNMENT_GRAPH_H_ + +#include +#include +#include + +#include "SyntaxTree.h" + +#include "Alignment.h" +#include "Options.h" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace GHKM +{ + +class Node; +class Subgraph; + +class AlignmentGraph +{ +public: + AlignmentGraph(const SyntaxTree *, + const std::vector &, + const Alignment &); + + ~AlignmentGraph(); + + Node *GetRoot() { + return m_root; + } + const std::vector &GetTargetNodes() { + return m_targetNodes; + } + + void ExtractMinimalRules(const Options &); + void ExtractComposedRules(const Options &); + +private: + // Disallow copying + AlignmentGraph(const AlignmentGraph &); + AlignmentGraph &operator=(const AlignmentGraph &); + + Node *CopyParseTree(const SyntaxTree *); + void ComputeFrontierSet(Node *, const Options &, std::set &) const; + bool IsFrontierNode(const Node &, const Options &) const; + void CalcComplementSpans(Node *); + void GetTargetTreeLeaves(Node *, std::vector &); + void AttachUnalignedSourceWords(); + Node *DetermineAttachmentPoint(int); + Subgraph ComputeMinimalFrontierGraphFragment(Node *, + const std::set &); + void ExtractComposedRules(Node *, const Options &); + + Node *m_root; + std::vector m_sourceNodes; + std::vector m_targetNodes; +}; + +} // namespace GHKM +} // namespace Syntax +} // namespace MosesTraining + +#endif diff --git a/mosesdecoder/phrase-extract/extract-ghkm/ComposedRule.cpp b/mosesdecoder/phrase-extract/extract-ghkm/ComposedRule.cpp new file mode 100644 index 0000000000000000000000000000000000000000..b4f6a6fcd46b2d93996753edeff0fa1fd6d5c79d --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-ghkm/ComposedRule.cpp @@ -0,0 +1,134 @@ +/*********************************************************************** + Moses - statistical machine translation system + Copyright (C) 2006-2011 University of Edinburgh + + This library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + This library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with this library; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +***********************************************************************/ + +#include "ComposedRule.h" + +#include +#include +#include + +#include "Node.h" +#include "Options.h" +#include "Subgraph.h" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace GHKM +{ + +ComposedRule::ComposedRule(const Subgraph &baseRule) + : m_baseRule(baseRule) + , m_depth(baseRule.GetDepth()) + , m_size(baseRule.GetSize()) + , m_nodeCount(baseRule.GetNodeCount()) +{ + const std::set &leaves = baseRule.GetLeaves(); + for (std::set::const_iterator p = leaves.begin(); + p != leaves.end(); ++p) { + if ((*p)->GetType() == TREE) { + m_openAttachmentPoints.push(*p); + } + } +} + +ComposedRule::ComposedRule(const ComposedRule &other, const Subgraph &rule, + int depth) + : m_baseRule(other.m_baseRule) + , m_attachedRules(other.m_attachedRules) + , m_openAttachmentPoints(other.m_openAttachmentPoints) + , m_depth(depth) + , m_size(other.m_size+rule.GetSize()) + , m_nodeCount(other.m_nodeCount+rule.GetNodeCount()-1) +{ + m_attachedRules.push_back(&rule); + m_openAttachmentPoints.pop(); +} + +const Node *ComposedRule::GetOpenAttachmentPoint() +{ + return m_openAttachmentPoints.empty() ? 0 : m_openAttachmentPoints.front(); +} + +void ComposedRule::CloseAttachmentPoint() +{ + assert(!m_openAttachmentPoints.empty()); + m_attachedRules.push_back(0); + m_openAttachmentPoints.pop(); +} + +ComposedRule *ComposedRule::AttemptComposition(const Subgraph &rule, + const Options &options) const +{ + // The smallest possible rule fragment should be rooted at a tree node. + // Note that this differs from the original GHKM definition. + assert(rule.GetRoot()->GetType() == TREE); + + // Check the node count of the proposed rule. + if (m_nodeCount+rule.GetNodeCount()-1 > options.maxNodes) { + return 0; + } + + // Check the size of the proposed rule. + if (m_size+rule.GetSize() > options.maxRuleSize) { + return 0; + } + + // Determine the depth of the proposed rule and test whether it exceeds the + // limit. + int attachmentPointDepth = 0; + const Node *n = rule.GetRoot(); + while (n != m_baseRule.GetRoot()) { + assert(n->GetParents().size() == 1); + n = n->GetParents()[0]; + ++attachmentPointDepth; + } + int newDepth = std::max(m_depth, attachmentPointDepth+rule.GetDepth()); + if (newDepth > options.maxRuleDepth) { + return 0; + } + + return new ComposedRule(*this, rule, newDepth); +} + +Subgraph ComposedRule::CreateSubgraph() +{ + std::set leaves; + const std::set &baseLeaves = m_baseRule.GetLeaves(); + size_t i = 0; + for (std::set::const_iterator p = baseLeaves.begin(); + p != baseLeaves.end(); ++p) { + const Node *baseLeaf = *p; + if (baseLeaf->GetType() == TREE && i < m_attachedRules.size()) { + const Subgraph *attachedRule = m_attachedRules[i++]; + if (attachedRule) { + leaves.insert(attachedRule->GetLeaves().begin(), + attachedRule->GetLeaves().end()); + continue; + } + } + leaves.insert(baseLeaf); + } + return Subgraph(m_baseRule.GetRoot(), leaves); +} + +} // namespace GHKM +} // namespace Syntax +} // namespace MosesTraining diff --git a/mosesdecoder/phrase-extract/extract-ghkm/ComposedRule.h b/mosesdecoder/phrase-extract/extract-ghkm/ComposedRule.h new file mode 100644 index 0000000000000000000000000000000000000000..9ff91029377aa9f6d8b5a6209368f786b4880c2d --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-ghkm/ComposedRule.h @@ -0,0 +1,75 @@ +/*********************************************************************** + Moses - statistical machine translation system + Copyright (C) 2006-2011 University of Edinburgh + + This library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + This library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with this library; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +***********************************************************************/ + +#pragma once +#ifndef EXTRACT_GHKM_COMPOSED_RULE_H_ +#define EXTRACT_GHKM_COMPOSED_RULE_H_ + +#include +#include + +#include "Subgraph.h" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace GHKM +{ + +class Node; +struct Options; + +class ComposedRule +{ +public: + // Form a 'trivial' ComposedRule from a single existing rule. + ComposedRule(const Subgraph &baseRule); + + // Returns the first open attachment point if any exist or 0 otherwise. + const Node *GetOpenAttachmentPoint(); + + // Close the first open attachment point without attaching a rule. + void CloseAttachmentPoint(); + + // Attempts to produce a new composed rule by attaching a given rule at the + // first open attachment point. This will fail if the proposed rule violates + // the constraints set in the Options object, in which case the function + // returns 0. + ComposedRule *AttemptComposition(const Subgraph &, const Options &) const; + + // Constructs a Subgraph object corresponding to the composed rule. + Subgraph CreateSubgraph(); + +private: + ComposedRule(const ComposedRule &, const Subgraph &, int); + + const Subgraph &m_baseRule; + std::vector m_attachedRules; + std::queue m_openAttachmentPoints; + int m_depth; + int m_size; + int m_nodeCount; +}; + +} // namespace GHKM +} // namespace Syntax +} // namespace MosesTraining + +#endif diff --git a/mosesdecoder/phrase-extract/extract-ghkm/ExtractGHKM.cpp b/mosesdecoder/phrase-extract/extract-ghkm/ExtractGHKM.cpp new file mode 100644 index 0000000000000000000000000000000000000000..1f91d616ba27f257b280d5cbd87997d1c04732b8 --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-ghkm/ExtractGHKM.cpp @@ -0,0 +1,902 @@ +/*********************************************************************** + Moses - statistical machine translation system + Copyright (C) 2006-2011 University of Edinburgh + + This library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + This library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with this library; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +***********************************************************************/ + +#include "ExtractGHKM.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "syntax-common/exception.h" +#include "syntax-common/xml_tree_parser.h" + +#include "InputFileStream.h" +#include "OutputFileStream.h" +#include "SyntaxNode.h" +#include "SyntaxNodeCollection.h" +#include "SyntaxTree.h" +#include "tables-core.h" +#include "XmlException.h" +#include "XmlTree.h" + +#include "Alignment.h" +#include "AlignmentGraph.h" +#include "Node.h" +#include "Options.h" +#include "PhraseOrientation.h" +#include "ScfgRule.h" +#include "ScfgRuleWriter.h" +#include "Span.h" +#include "StsgRule.h" +#include "StsgRuleWriter.h" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace GHKM +{ + +int ExtractGHKM::Main(int argc, char *argv[]) +{ + using Moses::InputFileStream; + using Moses::OutputFileStream; + + // Process command-line options. + Options options; + ProcessOptions(argc, argv, options); + + // Open input files. + // + // The GHKM algorithm is neutral about whether the model is string-to-tree or + // tree-to-string. This implementation assumes the model to be + // string-to-tree, but if the -t2s option is given then the source and target + // input files are switched prior to extraction and then the source and + // target of the extracted rules are switched on output. + std::string effectiveTargetFile = options.t2s ? options.sourceFile + : options.targetFile; + std::string effectiveSourceFile = options.t2s ? options.targetFile + : options.sourceFile; + InputFileStream targetStream(effectiveTargetFile); + InputFileStream sourceStream(effectiveSourceFile); + InputFileStream alignmentStream(options.alignmentFile); + + // Open output files. + OutputFileStream fwdExtractStream; + OutputFileStream invExtractStream; + OutputFileStream glueGrammarStream; + OutputFileStream targetUnknownWordStream; + OutputFileStream sourceUnknownWordStream; + OutputFileStream sourceLabelSetStream; + OutputFileStream unknownWordSoftMatchesStream; + + std::string fwdFileName = options.extractFile; + std::string invFileName = options.extractFile + std::string(".inv"); + if (options.gzOutput) { + fwdFileName += ".gz"; + invFileName += ".gz"; + } + OpenOutputFileOrDie(fwdFileName, fwdExtractStream); + OpenOutputFileOrDie(invFileName, invExtractStream); + + if (!options.glueGrammarFile.empty()) { + OpenOutputFileOrDie(options.glueGrammarFile, glueGrammarStream); + } + if (!options.targetUnknownWordFile.empty()) { + OpenOutputFileOrDie(options.targetUnknownWordFile, targetUnknownWordStream); + } + if (!options.sourceUnknownWordFile.empty()) { + OpenOutputFileOrDie(options.sourceUnknownWordFile, sourceUnknownWordStream); + } + if (!options.sourceLabelSetFile.empty()) { + if (!options.sourceLabels) { + Error("SourceLabels should be active if SourceLabelSet is supposed to be written to a file"); + } + OpenOutputFileOrDie(options.sourceLabelSetFile, sourceLabelSetStream); // note that this is not a global source label set if extraction is parallelized + } + if (!options.unknownWordSoftMatchesFile.empty()) { + OpenOutputFileOrDie(options.unknownWordSoftMatchesFile, unknownWordSoftMatchesStream); + } + + // Word count statistics for producing unknown word labels. + std::map targetWordCount; + std::map targetWordLabel; + + // Word count statistics for producing unknown word labels: source side. + std::map sourceWordCount; + std::map sourceWordLabel; + + std::string targetLine; + std::string sourceLine; + std::string alignmentLine; + Alignment alignment; + XmlTreeParser targetXmlTreeParser; + XmlTreeParser sourceXmlTreeParser; + ScfgRuleWriter scfgWriter(fwdExtractStream, invExtractStream, options); + StsgRuleWriter stsgWriter(fwdExtractStream, invExtractStream, options); + size_t lineNum = options.sentenceOffset; + while (true) { + std::getline(targetStream, targetLine); + std::getline(sourceStream, sourceLine); + std::getline(alignmentStream, alignmentLine); + + if (targetStream.eof() && sourceStream.eof() && alignmentStream.eof()) { + break; + } + + if (targetStream.eof() || sourceStream.eof() || alignmentStream.eof()) { + Error("Files must contain same number of lines"); + } + + ++lineNum; + + // Parse target tree. + if (targetLine.size() == 0) { + std::cerr << "skipping line " << lineNum << " with empty target tree\n"; + continue; + } + std::auto_ptr targetParseTree; + try { + targetParseTree = targetXmlTreeParser.Parse(targetLine); + assert(targetParseTree.get()); + } catch (const Exception &e) { + std::ostringstream oss; + oss << "Failed to parse target XML tree at line " << lineNum; + if (!e.msg().empty()) { + oss << ": " << e.msg(); + } + Error(oss.str()); + } + + // Read source tokens (and parse tree if using source labels). + std::vector sourceTokens; + std::auto_ptr sourceParseTree; + if (!options.sourceLabels) { + sourceTokens = ReadTokens(sourceLine); + } else { + try { + sourceParseTree = sourceXmlTreeParser.Parse(sourceLine); + assert(sourceParseTree.get()); + } catch (const Exception &e) { + std::ostringstream oss; + oss << "Failed to parse source XML tree at line " << lineNum; + if (!e.msg().empty()) { + oss << ": " << e.msg(); + } + Error(oss.str()); + } + sourceTokens = sourceXmlTreeParser.words(); + } + + // Read word alignments. + try { + ReadAlignment(alignmentLine, alignment); + } catch (const Exception &e) { + std::ostringstream oss; + oss << "Failed to read alignment at line " << lineNum << ": "; + oss << e.msg(); + Error(oss.str()); + } + if (alignment.size() == 0) { + std::cerr << "skipping line " << lineNum << " without alignment points\n"; + continue; + } + if (options.t2s) { + FlipAlignment(alignment); + } + + // Record word counts. + if (!options.targetUnknownWordFile.empty()) { + CollectWordLabelCounts(*targetParseTree, options, targetWordCount, + targetWordLabel); + } + + // Record word counts: source side. + if (options.sourceLabels && !options.sourceUnknownWordFile.empty()) { + CollectWordLabelCounts(*sourceParseTree, options, sourceWordCount, + sourceWordLabel); + } + + // Form an alignment graph from the target tree, source words, and + // alignment. + AlignmentGraph graph(targetParseTree.get(), sourceTokens, alignment); + + // Extract minimal rules, adding each rule to its root node's rule set. + graph.ExtractMinimalRules(options); + + // Extract composed rules. + if (!options.minimal) { + graph.ExtractComposedRules(options); + } + + // Initialize phrase orientation scoring object + PhraseOrientation phraseOrientation(sourceTokens.size(), + targetXmlTreeParser.words().size(), alignment); + + // Write the rules, subject to scope pruning. + const std::vector &targetNodes = graph.GetTargetNodes(); + for (std::vector::const_iterator p = targetNodes.begin(); + p != targetNodes.end(); ++p) { + + const std::vector &rules = (*p)->GetRules(); + + PhraseOrientation::REO_CLASS l2rOrientation=PhraseOrientation::REO_CLASS_UNKNOWN, r2lOrientation=PhraseOrientation::REO_CLASS_UNKNOWN; + if (options.phraseOrientation && !rules.empty()) { + int sourceSpanBegin = *((*p)->GetSpan().begin()); + int sourceSpanEnd = *((*p)->GetSpan().rbegin()); + l2rOrientation = phraseOrientation.GetOrientationInfo(sourceSpanBegin,sourceSpanEnd,PhraseOrientation::REO_DIR_L2R); + r2lOrientation = phraseOrientation.GetOrientationInfo(sourceSpanBegin,sourceSpanEnd,PhraseOrientation::REO_DIR_R2L); + // std::cerr << "span " << sourceSpanBegin << " " << sourceSpanEnd << std::endl; + // std::cerr << "phraseOrientation " << phraseOrientation.GetOrientationInfo(sourceSpanBegin,sourceSpanEnd) << std::endl; + } + + for (std::vector::const_iterator q = rules.begin(); + q != rules.end(); ++q) { + // STSG output. + if (options.stsg) { + StsgRule rule(**q); + if (rule.Scope() <= options.maxScope) { + stsgWriter.Write(rule); + } + continue; + } + // SCFG output. + ScfgRule *r = 0; + if (options.sourceLabels) { + r = new ScfgRule(**q, &sourceXmlTreeParser.node_collection()); + } else { + r = new ScfgRule(**q); + } + // TODO Can scope pruning be done earlier? + if (r->Scope() <= options.maxScope) { + scfgWriter.Write(*r,lineNum,false); + if (options.treeFragments) { + fwdExtractStream << " {{Tree "; + (*q)->PrintTree(fwdExtractStream); + fwdExtractStream << "}}"; + } + if (options.partsOfSpeech) { + fwdExtractStream << " {{POS"; + (*q)->PrintPartsOfSpeech(fwdExtractStream); + fwdExtractStream << "}}"; + } + if (options.phraseOrientation) { + fwdExtractStream << " {{Orientation "; + phraseOrientation.WriteOrientation(fwdExtractStream,l2rOrientation); + fwdExtractStream << " "; + phraseOrientation.WriteOrientation(fwdExtractStream,r2lOrientation); + fwdExtractStream << "}}"; + phraseOrientation.IncrementPriorCount(PhraseOrientation::REO_DIR_L2R,l2rOrientation,1); + phraseOrientation.IncrementPriorCount(PhraseOrientation::REO_DIR_R2L,r2lOrientation,1); + } + fwdExtractStream << std::endl; + invExtractStream << std::endl; + } + delete r; + } + } + } + + if (options.phraseOrientation) { + std::string phraseOrientationPriorsFileName = options.extractFile + std::string(".phraseOrientationPriors"); + OutputFileStream phraseOrientationPriorsStream; + OpenOutputFileOrDie(phraseOrientationPriorsFileName, phraseOrientationPriorsStream); + PhraseOrientation::WritePriorCounts(phraseOrientationPriorsStream); + } + + std::map sourceLabels; + if (options.sourceLabels && !options.sourceLabelSetFile.empty()) { + std::set extendedLabelSet = sourceXmlTreeParser.label_set(); + extendedLabelSet.insert("XLHS"); // non-matching label (left-hand side) + extendedLabelSet.insert("XRHS"); // non-matching label (right-hand side) + extendedLabelSet.insert("TOPLABEL"); // as used in the glue grammar + extendedLabelSet.insert("SOMELABEL"); // as used in the glue grammar + size_t index = 0; + for (std::set::const_iterator iter=extendedLabelSet.begin(); + iter!=extendedLabelSet.end(); ++iter, ++index) { + sourceLabels.insert(std::pair(*iter,index)); + } + WriteSourceLabelSet(sourceLabels, sourceLabelSetStream); + } + + std::set strippedTargetLabelSet; + std::map strippedTargetTopLabelSet; + if (options.stripBitParLabels && + (!options.glueGrammarFile.empty() || !options.unknownWordSoftMatchesFile.empty())) { + StripBitParLabels(targetXmlTreeParser.label_set(), + targetXmlTreeParser.top_label_set(), + strippedTargetLabelSet, strippedTargetTopLabelSet); + } + + if (!options.glueGrammarFile.empty()) { + if (options.stripBitParLabels) { + WriteGlueGrammar(strippedTargetLabelSet, strippedTargetTopLabelSet, sourceLabels, options, glueGrammarStream); + } else { + WriteGlueGrammar(targetXmlTreeParser.label_set(), + targetXmlTreeParser.top_label_set(), + sourceLabels, options, glueGrammarStream); + } + } + + if (!options.targetUnknownWordFile.empty()) { + WriteUnknownWordLabel(targetWordCount, targetWordLabel, options, targetUnknownWordStream); + } + + if (options.sourceLabels && !options.sourceUnknownWordFile.empty()) { + WriteUnknownWordLabel(sourceWordCount, sourceWordLabel, options, sourceUnknownWordStream, true); + } + + if (!options.unknownWordSoftMatchesFile.empty()) { + if (options.stripBitParLabels) { + WriteUnknownWordSoftMatches(strippedTargetLabelSet, unknownWordSoftMatchesStream); + } else { + WriteUnknownWordSoftMatches(targetXmlTreeParser.label_set(), + unknownWordSoftMatchesStream); + } + } + + return 0; +} + +void ExtractGHKM::ProcessOptions(int argc, char *argv[], + Options &options) const +{ + namespace po = boost::program_options; + namespace cls = boost::program_options::command_line_style; + + // Construct the 'top' of the usage message: the bit that comes before the + // options list. + std::ostringstream usageTop; + usageTop << "Usage: " << name() + << " [OPTION]... TARGET SOURCE ALIGNMENT EXTRACT\n\n" + << "SCFG rule extractor based on the GHKM algorithm described in\n" + << "Galley et al. (2004).\n\n" + << "Options"; + + // Construct the 'bottom' of the usage message. + std::ostringstream usageBottom; + usageBottom << "\nImplementation Notes:\n" + << "\nThe parse tree is assumed to contain part-of-speech preterminal nodes.\n" + << "\n" + << "For the composed rule constraints: rule depth is the " + "maximum distance from the\nrule's root node to a sink " + "node, not counting preterminal expansions or word\n" + "alignments. Rule size is the measure defined in DeNeefe " + "et al (2007): the\nnumber of non-part-of-speech, non-leaf " + "constituent labels in the target tree.\nNode count is the " + "number of target tree nodes (excluding target words).\n" + << "\n" + << "Scope pruning (Hopkins and Langmead, 2010) is applied to both minimal and\ncomposed rules.\n" + << "\n" + << "Unaligned source words are attached to the tree using the " + "following heuristic:\nif there are aligned source words to " + "both the left and the right of an unaligned\nsource word " + "then it is attached to the lowest common ancestor of its " + "nearest\nsuch left and right neighbours. Otherwise, it is " + "attached to the root of the\nparse tree.\n" + << "\n" + << "Unless the --AllowUnary option is given, unary rules containing no lexical\nsource items are eliminated using the method described in Chung et al. (2011).\nThe parsing algorithm used in Moses is unable to handle such rules.\n" + << "\n" + << "References:\n" + << "Galley, M., Hopkins, M., Knight, K., and Marcu, D. (2004)\n" + << "\"What's in a Translation Rule?\", In Proceedings of HLT/NAACL 2004.\n" + << "\n" + << "DeNeefe, S., Knight, K., Wang, W., and Marcu, D. (2007)\n" + << "\"What Can Syntax-Based MT Learn from Phrase-Based MT?\", In Proceedings of\nEMNLP-CoNLL 2007.\n" + << "\n" + << "Hopkins, M. and Langmead, G. (2010)\n" + << "\"SCFG Decoding Without Binarization\", In Proceedings of EMNLP 2010.\n" + << "\n" + << "Chung, T. and Fang, L. and Gildea, D. (2011)\n" + << "\"Issues Concerning Decoding with Synchronous Context-free Grammar\", In\nProceedings of ACL/HLT 2011."; + + // Declare the command line options that are visible to the user. + po::options_description visible(usageTop.str()); + visible.add_options() + //("help", "print this help message and exit") + ("AllowUnary", + "allow fully non-lexical unary rules") + ("ConditionOnTargetLHS", + "write target LHS instead of \"X\" as source LHS") + ("GlueGrammar", + po::value(&options.glueGrammarFile), + "write glue grammar to named file") + ("GZOutput", + "write gzipped extract files") + ("IncludeSentenceId", + "include sentence ID") + ("MaxNodes", + po::value(&options.maxNodes)->default_value(options.maxNodes), + "set maximum number of tree nodes for composed rules") + ("MaxRuleDepth", + po::value(&options.maxRuleDepth)->default_value(options.maxRuleDepth), + "set maximum depth for composed rules") + ("MaxRuleSize", + po::value(&options.maxRuleSize)->default_value(options.maxRuleSize), + "set maximum size for composed rules") + ("MaxScope", + po::value(&options.maxScope)->default_value(options.maxScope), + "set maximum allowed scope") + ("Minimal", + "extract minimal rules only") + ("PartsOfSpeech", + "output parts-of-speech as property (preterminals from the parse tree)") + ("PartsOfSpeechFactor", + "output parts-of-speech as factor (preterminals from the parse tree)") + ("PCFG", + "include score based on PCFG scores in target corpus") + ("PhraseOrientation", + "output phrase orientation information") + ("StripBitParLabels", + "strip suffix starting with a hyphen symbol (\"-\") from non-terminal labels") + ("STSG", + "output STSG rules (default is SCFG)") + ("T2S", + "enable tree-to-string rule extraction (string-to-tree is assumed by default)") + ("TreeFragments", + "output parse tree information") + ("SourceLabels", + "output source syntax label information") + ("SourceLabelSet", + po::value(&options.sourceLabelSetFile), + "write source syntax label set to named file") + ("SentenceOffset", + po::value(&options.sentenceOffset)->default_value(options.sentenceOffset), + "set sentence number offset if processing split corpus") + ("UnknownWordLabel", + po::value(&options.targetUnknownWordFile), + "write unknown word labels to named file") + ("SourceUnknownWordLabel", + po::value(&options.sourceUnknownWordFile), + "write source syntax unknown word labels to named file") + ("UnknownWordMinRelFreq", + po::value(&options.unknownWordMinRelFreq)->default_value( + options.unknownWordMinRelFreq), + "set minimum relative frequency for unknown word labels") + ("UnknownWordSoftMatches", + po::value(&options.unknownWordSoftMatchesFile), + "write dummy value to unknown word label file, and mappings from dummy value to other labels to named file") + ("UnknownWordUniform", + "write uniform weights to unknown word label file") + ("UnpairedExtractFormat", + "do not pair non-terminals in extract files") + ; + + // Declare the command line options that are hidden from the user + // (these are used as positional options). + po::options_description hidden("Hidden options"); + hidden.add_options() + ("TargetFile", + po::value(&options.targetFile), + "target file") + ("SourceFile", + po::value(&options.sourceFile), + "source file") + ("AlignmentFile", + po::value(&options.alignmentFile), + "alignment file") + ("ExtractFile", + po::value(&options.extractFile), + "extract file") + ; + + // Compose the full set of command-line options. + po::options_description cmdLineOptions; + cmdLineOptions.add(visible).add(hidden); + + // Register the positional options. + po::positional_options_description p; + p.add("TargetFile", 1); + p.add("SourceFile", 1); + p.add("AlignmentFile", 1); + p.add("ExtractFile", 1); + + // Process the command-line. + po::variables_map vm; + try { + po::store(po::command_line_parser(argc, argv).style(MosesOptionStyle()). + options(cmdLineOptions).positional(p).run(), vm); + po::notify(vm); + } catch (const std::exception &e) { + std::ostringstream msg; + msg << e.what() << "\n\n" << visible << usageBottom.str(); + Error(msg.str()); + } + + if (vm.count("help")) { + std::cout << visible << usageBottom.str() << std::endl; + std::exit(0); + } + + // Check all positional options were given. + if (!vm.count("TargetFile") || + !vm.count("SourceFile") || + !vm.count("AlignmentFile") || + !vm.count("ExtractFile")) { + std::ostringstream msg; + std::cerr << visible << usageBottom.str() << std::endl; + std::exit(1); + } + + // Process Boolean options. + if (vm.count("AllowUnary")) { + options.allowUnary = true; + } + if (vm.count("ConditionOnTargetLHS")) { + options.conditionOnTargetLhs = true; + } + if (vm.count("GZOutput")) { + options.gzOutput = true; + } + if (vm.count("IncludeSentenceId")) { + options.includeSentenceId = true; + } + if (vm.count("Minimal")) { + options.minimal = true; + } + if (vm.count("PartsOfSpeech")) { + options.partsOfSpeech = true; + } + if (vm.count("PartsOfSpeechFactor")) { + options.partsOfSpeechFactor = true; + } + if (vm.count("PCFG")) { + options.pcfg = true; + } + if (vm.count("PhraseOrientation")) { + options.phraseOrientation = true; + } + if (vm.count("StripBitParLabels")) { + options.stripBitParLabels = true; + } + if (vm.count("STSG")) { + options.stsg = true; + } + if (vm.count("T2S")) { + options.t2s = true; + } + if (vm.count("TreeFragments")) { + options.treeFragments = true; + } + if (vm.count("SourceLabels")) { + options.sourceLabels = true; + } + if (vm.count("UnknownWordUniform")) { + options.unknownWordUniform = true; + } + if (vm.count("UnpairedExtractFormat")) { + options.unpairedExtractFormat = true; + } + + // Workaround for extract-parallel issue. + if (options.sentenceOffset > 0) { + options.targetUnknownWordFile.clear(); + } + if (options.sentenceOffset > 0) { + options.sourceUnknownWordFile.clear(); + options.unknownWordSoftMatchesFile.clear(); + } +} + +std::vector ExtractGHKM::ReadTokens(const std::string &s) const +{ + std::vector tokens; + + std::string whitespace = " \t"; + + std::string::size_type begin = s.find_first_not_of(whitespace); + assert(begin != std::string::npos); + while (true) { + std::string::size_type end = s.find_first_of(whitespace, begin); + std::string token; + if (end == std::string::npos) { + token = s.substr(begin); + } else { + token = s.substr(begin, end-begin); + } + tokens.push_back(token); + if (end == std::string::npos) { + break; + } + begin = s.find_first_not_of(whitespace, end); + if (begin == std::string::npos) { + break; + } + } + + return tokens; +} + +void ExtractGHKM::WriteGlueGrammar( + const std::set &labelSet, + const std::map &topLabelSet, + const std::map &sourceLabels, + const Options &options, + std::ostream &out) const +{ + // choose a top label that is not already a label + std::string topLabel = "QQQQQQ"; + for(size_t i = 1; i <= topLabel.length(); i++) { + if (labelSet.find(topLabel.substr(0,i)) == labelSet.end() ) { + topLabel = topLabel.substr(0,i); + break; + } + } + + const size_t sourceLabelGlueTop = 0; + const size_t sourceLabelGlueX = 1; + const size_t sourceLabelSentenceStart = 2; + const size_t sourceLabelSentenceEnd = 3; +// const size_t partOfSpeechSentenceStart = 0; +// const size_t partOfSpeechSentenceEnd = 1; + +#ifndef BOS_ +#define BOS_ "" //Beginning of sentence symbol +#endif +#ifndef EOS_ +#define EOS_ "" //End of sentence symbol +#endif + + std::string sentenceStartSource = BOS_; + std::string sentenceEndSource = EOS_; + std::string sentenceStartTarget = BOS_; + std::string sentenceEndTarget = EOS_; + if (options.partsOfSpeech) { + sentenceStartTarget = sentenceStartTarget + "|" + BOS_; + sentenceEndTarget = sentenceEndTarget + "|" + EOS_; + } + if (options.partsOfSpeechFactor) { + sentenceStartTarget = sentenceStartTarget + "|" + BOS_; + sentenceEndTarget = sentenceEndTarget + "|" + EOS_; + } + + // basic rules + out << sentenceStartSource << " [X] ||| " << sentenceStartTarget << " [" << topLabel << "] ||| 1 ||| 0-0 ||| ||| |||"; + if (options.treeFragments) { + out << " {{Tree [" << topLabel << " [SSTART ]]}}"; + } +// if (options.partsOfSpeech) { +// out << " {{POS " << partOfSpeechSentenceStart << "}}"; +// } + if (options.sourceLabels) { + out << " {{SourceLabels 2 1 " << sourceLabelSentenceStart << " 1 1 " << sourceLabelGlueTop << " 1}}"; + } + if (options.phraseOrientation) { + out << " {{Orientation 1 1 0.5 0.5 1 1 0.5 0.5}}"; + } + out << std::endl; + + out << "[X][" << topLabel << "] " << sentenceEndSource << " [X] ||| [X][" << topLabel << "] " << sentenceEndTarget << " [" << topLabel << "] ||| 1 ||| 0-0 1-1 ||| ||| |||"; + if (options.treeFragments) { + out << " {{Tree [" << topLabel << " [" << topLabel << "] [SEND ]]}}"; + } +// if (options.partsOfSpeech) { +// out << " {{POS " << partOfSpeechSentenceEnd << "}}"; +// } + if (options.sourceLabels) { + out << " {{SourceLabels 4 1 " << sourceLabelSentenceStart << " " << sourceLabelGlueTop << " " << sourceLabelSentenceEnd << " 1 1 " << sourceLabelGlueTop << " 1}}"; + } + if (options.phraseOrientation) { + out << " {{Orientation 1 1 0.5 0.5 1 1 0.5 0.5}}"; + } + out << std::endl; + + // top rules + for (std::map::const_iterator i = topLabelSet.begin(); + i != topLabelSet.end(); ++i) { + out << sentenceStartSource << " [X][" << i->first << "] " << sentenceEndSource << " [X] ||| " << sentenceStartTarget << " [X][" << i->first << "] " << sentenceEndTarget << " [" << topLabel << "] ||| 1 ||| 0-0 1-1 2-2 ||| ||| |||"; + if (options.treeFragments) { + out << " {{Tree [" << topLabel << " [SSTART ] [" << i->first << "] [SEND ]]}}"; + } +// if (options.partsOfSpeech) { +// out << " {{POS " << partOfSpeechSentenceStart << " " << partOfSpeechSentenceEnd << "}}"; +// } + if (options.sourceLabels) { + out << " {{SourceLabels 4 1 " << sourceLabelSentenceStart << " " << sourceLabelGlueX << " " << sourceLabelSentenceEnd << " 1 1 " << sourceLabelGlueTop << " 1}}"; + } + if (options.phraseOrientation) { + out << " {{Orientation 1 1 0.5 0.5 1 1 0.5 0.5}}"; + } + out << std::endl; + } + + // glue rules + for(std::set::const_iterator i = labelSet.begin(); + i != labelSet.end(); i++ ) { + out << "[X][" << topLabel << "] [X][" << *i << "] [X] ||| [X][" << topLabel << "] [X][" << *i << "] [" << topLabel << "] ||| 2.718 ||| 0-0 1-1 ||| ||| |||"; + if (options.treeFragments) { + out << " {{Tree [" << topLabel << " ["<< topLabel << "] [" << *i << "]]}}"; + } + if (options.sourceLabels) { + out << " {{SourceLabels 3 1 " << sourceLabelGlueTop << " " << sourceLabelGlueX << " 1 1 " << sourceLabelGlueTop << " 1}}"; + } + if (options.phraseOrientation) { + out << " {{Orientation 1 1 0.5 0.5 1 1 0.5 0.5}}"; + } + out << std::endl; + } + + // glue rule for unknown word... + out << "[X][" << topLabel << "] [X][X] [X] ||| [X][" << topLabel << "] [X][X] [" << topLabel << "] ||| 2.718 ||| 0-0 1-1 ||| ||| |||"; + if (options.treeFragments) { + out << " {{Tree [" << topLabel << " [" << topLabel << "] [X]]}}"; + } + if (options.sourceLabels) { + out << " {{SourceLabels 3 1 " << sourceLabelGlueTop << " " << sourceLabelGlueX << " 1 1 " << sourceLabelGlueTop << " 1}}"; + } + if (options.phraseOrientation) { + out << " {{Orientation 1 1 0.5 0.5 1 1 0.5 0.5}}"; + } + out << std::endl; +} + +void ExtractGHKM::WriteSourceLabelSet( + const std::map &sourceLabels, + std::ostream &out) const +{ + out << sourceLabels.size() << std::endl; + for (std::map::const_iterator iter=sourceLabels.begin(); + iter!=sourceLabels.end(); ++iter) { + out << iter->first << " " << iter->second << std::endl; + } +} + +void ExtractGHKM::CollectWordLabelCounts( + SyntaxTree &root, + const Options &options, + std::map &wordCount, + std::map &wordLabel) +{ + for (SyntaxTree::ConstLeafIterator p(root); + p != SyntaxTree::ConstLeafIterator(); ++p) { + const SyntaxTree &leaf = *p; + const std::string &word = leaf.value().label; + const SyntaxTree *ancestor = leaf.parent(); + // If unary rule elimination is enabled and this word is at the end of a + // chain of unary rewrites, e.g. + // PN-SB -> NE -> word + // then record the constituent label at the top of the chain instead of + // the part-of-speech label. + while (!options.allowUnary && + ancestor->parent() && + ancestor->parent()->children().size() == 1) { + ancestor = ancestor->parent(); + } + const std::string &label = ancestor->value().label; + ++wordCount[word]; + wordLabel[word] = label; + } +} + +std::vector ExtractGHKM::ReadTokens(const SyntaxTree &root) const +{ + std::vector tokens; + for (SyntaxTree::ConstLeafIterator p(root); + p != SyntaxTree::ConstLeafIterator(); ++p) { + const SyntaxTree &leaf = *p; + const std::string &word = leaf.value().label; + tokens.push_back(word); + } + return tokens; +} + +void ExtractGHKM::WriteUnknownWordLabel( + const std::map &wordCount, + const std::map &wordLabel, + const Options &options, + std::ostream &out, + bool writeCounts) const +{ + if (!options.unknownWordSoftMatchesFile.empty()) { + out << "UNK 1" << std::endl; + return; + } + + std::map labelCount; + int total = 0; + for (std::map::const_iterator p = wordCount.begin(); + p != wordCount.end(); ++p) { + // Only consider singletons. + if (p->second == 1) { + std::map::const_iterator q = + wordLabel.find(p->first); + assert(q != wordLabel.end()); + if (options.stripBitParLabels) { + size_t pos = q->second.find('-'); + if (pos == std::string::npos) { + ++labelCount[q->second]; + } else { + ++labelCount[q->second.substr(0,pos)]; + } + } else { + ++labelCount[q->second]; + } + ++total; + } + } + if ( writeCounts ) { + for (std::map::const_iterator p = labelCount.begin(); + p != labelCount.end(); ++p) { + out << p->first << " " << p->second << std::endl; + } + } else { + for (std::map::const_iterator p = labelCount.begin(); + p != labelCount.end(); ++p) { + double ratio = static_cast(p->second) / static_cast(total); + if (ratio >= options.unknownWordMinRelFreq) { + float weight = options.unknownWordUniform ? 1.0f : ratio; + out << p->first << " " << weight << std::endl; + } + } + } +} + +void ExtractGHKM::WriteUnknownWordSoftMatches( + const std::set &labelSet, + std::ostream &out) const +{ + for (std::set::const_iterator p = labelSet.begin(); p != labelSet.end(); ++p) { + std::string label = *p; + out << "UNK " << label << std::endl; + } +} + +void ExtractGHKM::StripBitParLabels( + const std::set &labelSet, + const std::map &topLabelSet, + std::set &outLabelSet, + std::map &outTopLabelSet) const +{ + for (std::set::const_iterator it=labelSet.begin(); + it!=labelSet.end(); ++it) { + size_t pos = it->find('-'); + if (pos == std::string::npos) { + outLabelSet.insert(*it); + } else { + outLabelSet.insert(it->substr(0,pos)); + } + } + for (std::map::const_iterator it=topLabelSet.begin(); + it!=topLabelSet.end(); ++it) { + size_t pos = it->first.find('-'); + std::string stripped; + if (pos == std::string::npos) { + stripped = it->first; + } else { + stripped = it->first.substr(0,pos); + } + std::map::iterator found=outTopLabelSet.find(stripped); + if (found != outTopLabelSet.end()) { + found->second += it->second; + } else { + outTopLabelSet.insert(std::pair(stripped,it->second)); + } + } +} + +} // namespace GHKM +} // namespace Syntax +} // namespace MosesTraining diff --git a/mosesdecoder/phrase-extract/extract-ghkm/ExtractGHKM.h b/mosesdecoder/phrase-extract/extract-ghkm/ExtractGHKM.h new file mode 100644 index 0000000000000000000000000000000000000000..170de7ae9c6b13627da6d314f1dd716587b1e641 --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-ghkm/ExtractGHKM.h @@ -0,0 +1,82 @@ +/*********************************************************************** + Moses - statistical machine translation system + Copyright (C) 2006-2011 University of Edinburgh + + This library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + This library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with this library; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +***********************************************************************/ + +#pragma once + +#include +#include +#include +#include +#include + +#include "OutputFileStream.h" +#include "SyntaxTree.h" + +#include "syntax-common/tool.h" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace GHKM +{ + +struct Options; + +class ExtractGHKM : public Tool +{ +public: + ExtractGHKM() : Tool("extract-ghkm") {} + + virtual int Main(int argc, char *argv[]); + +private: + void RecordTreeLabels(const SyntaxTree &, std::set &); + void CollectWordLabelCounts(SyntaxTree &, + const Options &, + std::map &, + std::map &); + void WriteUnknownWordLabel(const std::map &, + const std::map &, + const Options &, + std::ostream &, + bool writeCounts=false) const; + void WriteUnknownWordSoftMatches(const std::set &, + std::ostream &) const; + void WriteGlueGrammar(const std::set &, + const std::map &, + const std::map &, + const Options &, + std::ostream &) const; + void WriteSourceLabelSet(const std::map &, + std::ostream &) const; + void StripBitParLabels(const std::set &labelSet, + const std::map &topLabelSet, + std::set &outLabelSet, + std::map &outTopLabelSet) const; + + std::vector ReadTokens(const std::string &) const; + std::vector ReadTokens(const SyntaxTree &root) const; + + void ProcessOptions(int, char *[], Options &) const; +}; + +} // namespace GHKM +} // namespace Syntax +} // namespace MosesTraining diff --git a/mosesdecoder/phrase-extract/extract-ghkm/Jamfile b/mosesdecoder/phrase-extract/extract-ghkm/Jamfile new file mode 100644 index 0000000000000000000000000000000000000000..4692937de3c980bb75adae10c7224f72e41d7949 --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-ghkm/Jamfile @@ -0,0 +1 @@ +exe extract-ghkm : [ glob *.cpp ] ..//syntax-common ..//deps ../..//boost_iostreams ../..//boost_program_options ../..//z : .. ; diff --git a/mosesdecoder/phrase-extract/extract-ghkm/Main.cpp b/mosesdecoder/phrase-extract/extract-ghkm/Main.cpp new file mode 100644 index 0000000000000000000000000000000000000000..f7a2173fbd677c29105ecbdcc52c2d942047541f --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-ghkm/Main.cpp @@ -0,0 +1,26 @@ +/*********************************************************************** + Moses - statistical machine translation system + Copyright (C) 2006-2011 University of Edinburgh + + This library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + This library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with this library; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +***********************************************************************/ + +#include "ExtractGHKM.h" + +int main(int argc, char *argv[]) +{ + MosesTraining::Syntax::GHKM::ExtractGHKM tool; + return tool.Main(argc, argv); +} diff --git a/mosesdecoder/phrase-extract/extract-ghkm/Node.cpp b/mosesdecoder/phrase-extract/extract-ghkm/Node.cpp new file mode 100644 index 0000000000000000000000000000000000000000..382fda99645de5d96b8b7609e7e6e785dcf13c30 --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-ghkm/Node.cpp @@ -0,0 +1,76 @@ +/*********************************************************************** + Moses - statistical machine translation system + Copyright (C) 2006-2011 University of Edinburgh + + This library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + This library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with this library; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +***********************************************************************/ + +#include "Node.h" + +#include "Subgraph.h" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace GHKM +{ + +Node::~Node() +{ + for (std::vector::const_iterator p(m_rules.begin()); + p != m_rules.end(); ++p) { + delete *p; + } +} + +bool Node::IsPreterminal() const +{ + return (m_type == TREE + && m_children.size() == 1 + && m_children[0]->m_type == TARGET); +} + +void Node::PropagateIndex(int index) +{ + m_span.insert(index); + for (std::vector::const_iterator p(m_parents.begin()); + p != m_parents.end(); ++p) { + (*p)->PropagateIndex(index); + } +} + +std::vector Node::GetTargetWords() const +{ + std::vector targetWords; + GetTargetWords(targetWords); + return targetWords; +} + +void Node::GetTargetWords(std::vector &targetWords) const +{ + if (m_type == TARGET) { + targetWords.push_back(m_label); + } else { + for (std::vector::const_iterator p(m_children.begin()); + p != m_children.end(); ++p) { + (*p)->GetTargetWords(targetWords); + } + } +} + +} // namespace GHKM +} // namespace Syntax +} // namespace MosesTraining diff --git a/mosesdecoder/phrase-extract/extract-ghkm/Node.h b/mosesdecoder/phrase-extract/extract-ghkm/Node.h new file mode 100644 index 0000000000000000000000000000000000000000..81f4a46b9e523540b5cc7725bfe582102e093f1a --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-ghkm/Node.h @@ -0,0 +1,223 @@ +/*********************************************************************** + Moses - statistical machine translation system + Copyright (C) 2006-2011 University of Edinburgh + + This library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + This library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with this library; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +***********************************************************************/ + +#pragma once +#ifndef EXTRACT_GHKM_NODE_H_ +#define EXTRACT_GHKM_NODE_H_ + +#include +#include +#include +#include + +#include "Span.h" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace GHKM +{ + +class Subgraph; + +enum NodeType { SOURCE, TARGET, TREE }; + +class Node +{ +public: + Node(const std::string &label, NodeType type) + : m_label(label) + , m_type(type) + , m_pcfgScore(0.0f) {} + + ~Node(); + + const std::string &GetLabel() const { + return m_label; + } + NodeType GetType() const { + return m_type; + } + const std::vector &GetChildren() const { + return m_children; + } + const std::vector &GetParents() const { + return m_parents; + } + float GetPcfgScore() const { + return m_pcfgScore; + } + const Span &GetSpan() const { + return m_span; + } + const Span &GetComplementSpan() const { + return m_complementSpan; + } + const std::vector &GetRules() const { + return m_rules; + } + + void SetChildren(const std::vector &c) { + m_children = c; + } + void SetParents(const std::vector &p) { + m_parents = p; + } + void SetPcfgScore(float s) { + m_pcfgScore = s; + } + void SetSpan(const Span &s) { + m_span = s; + } + void SetComplementSpan(const Span &cs) { + m_complementSpan = cs; + } + + void AddChild(Node *c) { + m_children.push_back(c); + } + void AddParent(Node *p) { + m_parents.push_back(p); + } + void AddRule(const Subgraph *s) { + m_rules.push_back(s); + } + + bool IsSink() const { + return m_children.empty(); + } + bool IsPreterminal() const; + + void PropagateIndex(int); + + std::vector GetTargetWords() const; + + // Gets the path from this node's parent to the root. This node is + // required to be part of the original parse tree (i.e. not a source word, + // which can have multiple parents). + template + void GetTreeAncestors(OutputIterator result, bool includeSelf=false); + + // Returns the lowest common ancestor given a sequence of nodes belonging to + // the target tree. + template + static Node *LowestCommonAncestor(InputIterator first, InputIterator last); + +private: + // Disallow copying + Node(const Node &); + Node &operator=(const Node &); + + void GetTargetWords(std::vector &) const; + + std::string m_label; + NodeType m_type; + std::vector m_children; + std::vector m_parents; + float m_pcfgScore; + Span m_span; + Span m_complementSpan; + std::vector m_rules; +}; + +template +void Node::GetTreeAncestors(OutputIterator result, bool includeSelf) +{ + // This function assumes the node is part of the parse tree. + assert(m_type == TARGET || m_type == TREE); + + if (includeSelf) { + *result++ = this; + } + + Node *ancestor = !(m_parents.empty()) ? m_parents[0] : 0; + while (ancestor != 0) { + *result++ = ancestor; + ancestor = !(ancestor->m_parents.empty()) ? ancestor->m_parents[0] : 0; + } +} + +template +Node *Node::LowestCommonAncestor(InputIterator first, InputIterator last) +{ + // Check for an empty sequence. + if (first == last) { + return 0; + } + + // Check for the case that the sequence contains only one distinct node. + // Also check that every node belongs to the target tree. + InputIterator p = first; + Node *lca = *p++; + for (; p != last; ++p) { + Node *node = *p; + assert(node->m_type != SOURCE); + if (node != lca) { + lca = 0; + } + } + if (lca) { + return lca; + } + + // Now construct an ancestor path for each node, from itself to the root. + size_t minPathLength = 0; + std::vector > paths; + for (p = first; p != last; ++p) { + paths.resize(paths.size()+1); + (*p)->GetTreeAncestors(std::back_inserter(paths.back()), true); + size_t pathLength = paths.back().size(); + assert(pathLength > 0); + if (paths.size() == 1 || pathLength < minPathLength) { + minPathLength = pathLength; + } + } + + // Search for the start of the longest common suffix by working forward from + // the the earliest possible starting point to the root. + for (size_t i = 0; i < minPathLength; ++i) { + bool match = true; + for (size_t j = 0; j < paths.size(); ++j) { + size_t index = paths[j].size() - minPathLength + i; + assert(index >= 0); + assert(index < paths[j].size()); + if (j == 0) { + lca = paths[j][index]; + assert(lca); + } else if (lca != paths[j][index]) { + match = false; + break; + } + } + if (match) { + return lca; + } + } + + // A lowest common ancestor should have been found. + assert(false); + return 0; +} + +} // namespace GHKM +} // namespace Syntax +} // namespace MosesTraining + +#endif diff --git a/mosesdecoder/phrase-extract/extract-ghkm/Options.h b/mosesdecoder/phrase-extract/extract-ghkm/Options.h new file mode 100644 index 0000000000000000000000000000000000000000..4294698838f2354c6edd16583aa3b918453bd0bf --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-ghkm/Options.h @@ -0,0 +1,95 @@ +/*********************************************************************** + Moses - statistical machine translation system + Copyright (C) 2006-2011 University of Edinburgh + + This library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + This library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with this library; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +***********************************************************************/ + +#pragma once + +#include + +namespace MosesTraining +{ +namespace Syntax +{ +namespace GHKM +{ + +struct Options { +public: + Options() + : allowUnary(false) + , conditionOnTargetLhs(false) + , gzOutput(false) + , includeSentenceId(false) + , maxNodes(15) + , maxRuleDepth(3) + , maxRuleSize(3) + , maxScope(3) + , minimal(false) + , partsOfSpeech(false) + , partsOfSpeechFactor(false) + , pcfg(false) + , phraseOrientation(false) + , sentenceOffset(0) + , sourceLabels(false) + , stripBitParLabels(false) + , stsg(false) + , t2s(false) + , treeFragments(false) + , unknownWordMinRelFreq(0.03f) + , unknownWordUniform(false) + , unpairedExtractFormat(false) {} + + // Positional options + std::string targetFile; + std::string sourceFile; + std::string alignmentFile; + std::string extractFile; + + // All other options + bool allowUnary; + bool conditionOnTargetLhs; + std::string glueGrammarFile; + bool gzOutput; + bool includeSentenceId; + int maxNodes; + int maxRuleDepth; + int maxRuleSize; + int maxScope; + bool minimal; + bool partsOfSpeech; + bool partsOfSpeechFactor; + bool pcfg; + bool phraseOrientation; + int sentenceOffset; + bool sourceLabels; + std::string sourceLabelSetFile; + std::string sourceUnknownWordFile; + bool stripBitParLabels; + bool stsg; + bool t2s; + std::string targetUnknownWordFile; + bool treeFragments; + float unknownWordMinRelFreq; + std::string unknownWordSoftMatchesFile; + bool unknownWordUniform; + bool unpairedExtractFormat; +}; + +} // namespace GHKM +} // namespace Syntax +} // namespace MosesTraining diff --git a/mosesdecoder/phrase-extract/extract-ghkm/Rule.cpp b/mosesdecoder/phrase-extract/extract-ghkm/Rule.cpp new file mode 100644 index 0000000000000000000000000000000000000000..b4b59f8e32f3d467c1f8b7a71eede85fcf75bd7e --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-ghkm/Rule.cpp @@ -0,0 +1,44 @@ +#include "Rule.h" + +#include "Node.h" +#include "Subgraph.h" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace GHKM +{ + +int Rule::Scope(const std::vector &symbols) +{ + int scope = 0; + bool predIsNonTerm = false; + if (symbols[0].GetType() == NonTerminal) { + ++scope; + predIsNonTerm = true; + } + for (std::size_t i = 1; i < symbols.size(); ++i) { + bool isNonTerm = symbols[i].GetType() == NonTerminal; + if (isNonTerm && predIsNonTerm) { + ++scope; + } + predIsNonTerm = isNonTerm; + } + if (predIsNonTerm) { + ++scope; + } + return scope; +} + +bool Rule::PartitionOrderComp(const Node *a, const Node *b) +{ + const Span &aSpan = a->GetSpan(); + const Span &bSpan = b->GetSpan(); + assert(!aSpan.empty() && !bSpan.empty()); + return *(aSpan.begin()) < *(bSpan.begin()); +} + +} // namespace GHKM +} // namespace Syntax +} // namespace MosesTraining diff --git a/mosesdecoder/phrase-extract/extract-ghkm/Rule.h b/mosesdecoder/phrase-extract/extract-ghkm/Rule.h new file mode 100644 index 0000000000000000000000000000000000000000..5317be7c8b15f5dcaaca3e3186c5c01522682dbd --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-ghkm/Rule.h @@ -0,0 +1,62 @@ +#pragma once +#ifndef EXTRACT_GHKM_RULE_H_ +#define EXTRACT_GHKM_RULE_H_ + +#include +#include + +#include "Alignment.h" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace GHKM +{ + +class Node; + +enum SymbolType { Terminal, NonTerminal }; + +class Symbol +{ +public: + Symbol(const std::string &v, SymbolType t) : m_value(v) , m_type(t) {} + + const std::string &GetValue() const { + return m_value; + } + SymbolType GetType() const { + return m_type; + } + +private: + std::string m_value; + SymbolType m_type; +}; + +// Base class for ScfgRule and StsgRule. +class Rule +{ +public: + virtual ~Rule() {} + + const Alignment &GetAlignment() const { + return m_alignment; + } + + virtual int Scope() const = 0; + +protected: + static bool PartitionOrderComp(const Node *, const Node *); + + static int Scope(const std::vector&); + + Alignment m_alignment; +}; + +} // namespace GHKM +} // namespace Syntax +} // namespace MosesTraining + +#endif diff --git a/mosesdecoder/phrase-extract/extract-ghkm/ScfgRule.cpp b/mosesdecoder/phrase-extract/extract-ghkm/ScfgRule.cpp new file mode 100644 index 0000000000000000000000000000000000000000..e26b17a871c87990367b0581d9aa848ad3dd69c0 --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-ghkm/ScfgRule.cpp @@ -0,0 +1,203 @@ +/*********************************************************************** + Moses - statistical machine translation system + Copyright (C) 2006-2011 University of Edinburgh + + This library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + This library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with this library; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +***********************************************************************/ + +#include "ScfgRule.h" + +#include + +#include "Node.h" +#include "Subgraph.h" +#include "SyntaxNode.h" +#include "SyntaxNodeCollection.h" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace GHKM +{ + +ScfgRule::ScfgRule(const Subgraph &fragment, + const SyntaxNodeCollection *sourceNodeCollection) + : m_graphFragment(fragment) + , m_sourceLHS("X", NonTerminal) + , m_targetLHS(fragment.GetRoot()->GetLabel(), NonTerminal) + , m_pcfgScore(fragment.GetPcfgScore()) + , m_hasSourceLabels(sourceNodeCollection) +{ + + // Source RHS + + const std::set &leaves = fragment.GetLeaves(); + + std::vector sourceRHSNodes; + sourceRHSNodes.reserve(leaves.size()); + for (std::set::const_iterator p(leaves.begin()); + p != leaves.end(); ++p) { + const Node &leaf = **p; + if (!leaf.GetSpan().empty()) { + sourceRHSNodes.push_back(&leaf); + } + } + + std::sort(sourceRHSNodes.begin(), sourceRHSNodes.end(), PartitionOrderComp); + + // Build a mapping from target nodes to source-order indices, so that we + // can construct the Alignment object later. + std::map > sourceOrder; + + m_sourceRHS.reserve(sourceRHSNodes.size()); + m_numberOfNonTerminals = 0; + int srcIndex = 0; + for (std::vector::const_iterator p(sourceRHSNodes.begin()); + p != sourceRHSNodes.end(); ++p, ++srcIndex) { + const Node &sinkNode = **p; + if (sinkNode.GetType() == TREE) { + m_sourceRHS.push_back(Symbol("X", NonTerminal)); + sourceOrder[&sinkNode].push_back(srcIndex); + ++m_numberOfNonTerminals; + } else { + assert(sinkNode.GetType() == SOURCE); + m_sourceRHS.push_back(Symbol(sinkNode.GetLabel(), Terminal)); + // Add all aligned target words to the sourceOrder map + const std::vector &parents(sinkNode.GetParents()); + for (std::vector::const_iterator q(parents.begin()); + q != parents.end(); ++q) { + if ((*q)->GetType() == TARGET) { + sourceOrder[*q].push_back(srcIndex); + } + } + } + if (sourceNodeCollection) { + // Source syntax label + PushSourceLabel(sourceNodeCollection,&sinkNode,"XRHS"); + } + } + + // Target RHS + alignment + + std::vector targetLeaves; + fragment.GetTargetLeaves(targetLeaves); + + m_alignment.reserve(targetLeaves.size()); // might be too much but that's OK + m_targetRHS.reserve(targetLeaves.size()); + + for (std::vector::const_iterator p(targetLeaves.begin()); + p != targetLeaves.end(); ++p) { + const Node &leaf = **p; + if (leaf.GetSpan().empty()) { + // The node doesn't cover any source words, so we can only add + // terminals to the target RHS (not a non-terminal). + std::vector targetWords(leaf.GetTargetWords()); + for (std::vector::const_iterator q(targetWords.begin()); + q != targetWords.end(); ++q) { + m_targetRHS.push_back(Symbol(*q, Terminal)); + } + } else if (leaf.GetType() == SOURCE) { + // Do nothing + } else { + SymbolType type = (leaf.GetType() == TREE) ? NonTerminal : Terminal; + m_targetRHS.push_back(Symbol(leaf.GetLabel(), type)); + + int tgtIndex = m_targetRHS.size()-1; + std::map >::iterator q(sourceOrder.find(&leaf)); + assert(q != sourceOrder.end()); + std::vector &sourceNodes = q->second; + for (std::vector::iterator r(sourceNodes.begin()); + r != sourceNodes.end(); ++r) { + int srcIndex = *r; + m_alignment.push_back(std::make_pair(srcIndex, tgtIndex)); + } + } + } + + if (sourceNodeCollection) { + // Source syntax label for root node (if sourceNodeCollection available) + PushSourceLabel(sourceNodeCollection,fragment.GetRoot(),"XLHS"); + // All non-terminal spans (including the LHS) should have obtained a label + // (a source-side syntactic constituent label if the span matches, "XLHS" otherwise) +// assert(m_sourceLabels.size() == m_numberOfNonTerminals+1); + } +} + +void ScfgRule::PushSourceLabel(const SyntaxNodeCollection *sourceNodeCollection, + const Node *node, + const std::string &nonMatchingLabel) +{ + ContiguousSpan span = Closure(node->GetSpan()); + if (sourceNodeCollection->HasNode(span.first,span.second)) { // does a source constituent match the span? + std::vector sourceLabels = + sourceNodeCollection->GetNodes(span.first,span.second); + if (!sourceLabels.empty()) { + // store the topmost matching label from the source syntax tree + m_sourceLabels.push_back(sourceLabels.back()->label); + } + } else { + // no matching source-side syntactic constituent: store nonMatchingLabel + m_sourceLabels.push_back(nonMatchingLabel); + } +} + +// TODO: rather implement the method external to ScfgRule +void ScfgRule::UpdateSourceLabelCoocCounts(std::map< std::string, std::map* > &coocCounts, float count) const +{ + std::map sourceToTargetNTMap; + std::map targetToSourceNTMap; + + for (Alignment::const_iterator p(m_alignment.begin()); + p != m_alignment.end(); ++p) { + if ( m_sourceRHS[p->first].GetType() == NonTerminal ) { + assert(m_targetRHS[p->second].GetType() == NonTerminal); + sourceToTargetNTMap[p->first] = p->second; + } + } + + size_t sourceIndex = 0; + size_t sourceNonTerminalIndex = 0; + for (std::vector::const_iterator p=m_sourceRHS.begin(); + p != m_sourceRHS.end(); ++p, ++sourceIndex) { + if ( p->GetType() == NonTerminal ) { + const std::string &sourceLabel = m_sourceLabels[sourceNonTerminalIndex]; + int targetIndex = sourceToTargetNTMap[sourceIndex]; + const std::string &targetLabel = m_targetRHS[targetIndex].GetValue(); + ++sourceNonTerminalIndex; + + std::map* countMap = NULL; + std::map< std::string, std::map* >::iterator iter = coocCounts.find(sourceLabel); + if ( iter == coocCounts.end() ) { + std::map *newCountMap = new std::map(); + std::pair< std::map< std::string, std::map* >::iterator, bool > inserted = + coocCounts.insert( std::pair< std::string, std::map* >(sourceLabel, newCountMap) ); + assert(inserted.second); + countMap = (inserted.first)->second; + } else { + countMap = iter->second; + } + std::pair< std::map::iterator, bool > inserted = + countMap->insert( std::pair< std::string,float>(targetLabel, count) ); + if ( !inserted.second ) { + (inserted.first)->second += count; + } + } + } +} + +} // namespace GHKM +} // namespace Syntax +} // namespace MosesTraining diff --git a/mosesdecoder/phrase-extract/extract-ghkm/ScfgRule.h b/mosesdecoder/phrase-extract/extract-ghkm/ScfgRule.h new file mode 100644 index 0000000000000000000000000000000000000000..c0ff483628588a60431fc5ac571e5382443da042 --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-ghkm/ScfgRule.h @@ -0,0 +1,99 @@ +/*********************************************************************** + Moses - statistical machine translation system + Copyright (C) 2006-2011 University of Edinburgh + + This library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + This library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with this library; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +***********************************************************************/ + +#pragma once + +#include +#include +#include +#include +#include + +#include "Alignment.h" +#include "Rule.h" +#include "SyntaxNodeCollection.h" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace GHKM +{ + +class Node; +class Subgraph; + +class ScfgRule : public Rule +{ +public: + ScfgRule(const Subgraph &fragment, + const SyntaxNodeCollection *sourceNodeCollection = 0); + + const Subgraph &GetGraphFragment() const { + return m_graphFragment; + } + const Symbol &GetSourceLHS() const { + return m_sourceLHS; + } + const Symbol &GetTargetLHS() const { + return m_targetLHS; + } + const std::vector &GetSourceRHS() const { + return m_sourceRHS; + } + const std::vector &GetTargetRHS() const { + return m_targetRHS; + } + float GetPcfgScore() const { + return m_pcfgScore; + } + bool HasSourceLabels() const { + return m_hasSourceLabels; + } + void PrintSourceLabels(std::ostream &out) const { + for (std::vector::const_iterator it = m_sourceLabels.begin(); + it != m_sourceLabels.end(); ++it) { + out << " " << (*it); + } + } + void UpdateSourceLabelCoocCounts(std::map< std::string, std::map* > &coocCounts, + float count) const; + + int Scope() const { + return Rule::Scope(m_sourceRHS); + } + +private: + void PushSourceLabel(const SyntaxNodeCollection *sourceNodeCollection, + const Node *node, const std::string &nonMatchingLabel); + + const Subgraph& m_graphFragment; + Symbol m_sourceLHS; + Symbol m_targetLHS; + std::vector m_sourceRHS; + std::vector m_targetRHS; + float m_pcfgScore; + bool m_hasSourceLabels; + std::vector m_sourceLabels; + unsigned m_numberOfNonTerminals; +}; + +} // namespace GHKM +} // namespace Syntax +} // namespace MosesTraining diff --git a/mosesdecoder/phrase-extract/extract-ghkm/ScfgRuleWriter.cpp b/mosesdecoder/phrase-extract/extract-ghkm/ScfgRuleWriter.cpp new file mode 100644 index 0000000000000000000000000000000000000000..e5cbc1d9914c2fb3c79ca7f00799930ca6894286 --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-ghkm/ScfgRuleWriter.cpp @@ -0,0 +1,235 @@ +/*********************************************************************** + Moses - statistical machine translation system + Copyright (C) 2006-2011 University of Edinburgh + + This library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + This library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with this library; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +***********************************************************************/ + +#include "ScfgRuleWriter.h" + +#include +#include +#include +#include +#include +#include + +#include "Alignment.h" +#include "Options.h" +#include "ScfgRule.h" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace GHKM +{ + +void ScfgRuleWriter::Write(const ScfgRule &rule, size_t lineNum, bool printEndl) +{ + std::ostringstream sourceSS; + std::ostringstream targetSS; + + if (m_options.unpairedExtractFormat) { + WriteUnpairedFormat(rule, sourceSS, targetSS); + } else { + WriteStandardFormat(rule, sourceSS, targetSS); + } + + // Write the rule to the forward and inverse extract files. + if (m_options.t2s) { + // If model is tree-to-string then flip the source and target. + m_fwd << targetSS.str() << " ||| " << sourceSS.str() << " |||"; + m_inv << sourceSS.str() << " ||| " << targetSS.str() << " |||"; + } else { + m_fwd << sourceSS.str() << " ||| " << targetSS.str() << " |||"; + m_inv << targetSS.str() << " ||| " << sourceSS.str() << " |||"; + } + + const Alignment &alignment = rule.GetAlignment(); + for (Alignment::const_iterator p = alignment.begin(); + p != alignment.end(); ++p) { + if (m_options.t2s) { + // If model is tree-to-string then flip the source and target. + m_fwd << " " << p->second << "-" << p->first; + m_inv << " " << p->first << "-" << p->second; + } else { + m_fwd << " " << p->first << "-" << p->second; + m_inv << " " << p->second << "-" << p->first; + } + } + + if (m_options.includeSentenceId) { + if (m_options.t2s) { + m_inv << " ||| " << lineNum; + } else { + m_fwd << " ||| " << lineNum; + } + } + + // Write a count of 1. + m_fwd << " ||| 1"; + m_inv << " ||| 1"; + + // Write the PCFG score (if requested). + if (m_options.pcfg) { + m_fwd << " ||| " << std::exp(rule.GetPcfgScore()); + } + + m_fwd << " |||"; + + if (m_options.sourceLabels && rule.HasSourceLabels()) { + m_fwd << " {{SourceLabels"; + rule.PrintSourceLabels(m_fwd); + m_fwd << "}}"; + } + + if (printEndl) { + m_fwd << std::endl; + m_inv << std::endl; + } +} + +void ScfgRuleWriter::WriteStandardFormat(const ScfgRule &rule, + std::ostream &sourceSS, + std::ostream &targetSS) +{ + const std::vector &sourceRHS = rule.GetSourceRHS(); + const std::vector &targetRHS = rule.GetTargetRHS(); + + std::map sourceToTargetNTMap; + std::map targetToSourceNTMap; + + const Alignment &alignment = rule.GetAlignment(); + + for (Alignment::const_iterator p(alignment.begin()); + p != alignment.end(); ++p) { + if (sourceRHS[p->first].GetType() == NonTerminal) { + assert(targetRHS[p->second].GetType() == NonTerminal); + sourceToTargetNTMap[p->first] = p->second; + targetToSourceNTMap[p->second] = p->first; + } + } + + // If parts-of-speech as a factor requested: retrieve preterminals from graph fragment + std::vector partsOfSpeech; + if (m_options.partsOfSpeechFactor) { + const Subgraph &graphFragment = rule.GetGraphFragment(); + graphFragment.GetPartsOfSpeech(partsOfSpeech); + } + + // Write the source side of the rule to sourceSS. + int i = 0; + for (std::vector::const_iterator p(sourceRHS.begin()); + p != sourceRHS.end(); ++p, ++i) { + WriteSymbol(*p, sourceSS); + if (p->GetType() == NonTerminal) { + int targetIndex = sourceToTargetNTMap[i]; + WriteSymbol(targetRHS[targetIndex], sourceSS); + } + sourceSS << " "; + } + if (m_options.conditionOnTargetLhs) { + WriteSymbol(rule.GetTargetLHS(), sourceSS); + } else { + WriteSymbol(rule.GetSourceLHS(), sourceSS); + } + + // Write the target side of the rule to targetSS. + i = 0; + int targetTerminalIndex = 0; + for (std::vector::const_iterator p(targetRHS.begin()); + p != targetRHS.end(); ++p, ++i) { + if (p->GetType() == NonTerminal) { + int sourceIndex = targetToSourceNTMap[i]; + WriteSymbol(sourceRHS[sourceIndex], targetSS); + } + WriteSymbol(*p, targetSS); + // If parts-of-speech as a factor requested: write part-of-speech + if (m_options.partsOfSpeechFactor && (p->GetType() != NonTerminal)) { + assert(targetTerminalIndex &sourceRHS = rule.GetSourceRHS(); + const std::vector &targetRHS = rule.GetTargetRHS(); + + // If parts-of-speech as a factor requested: retrieve preterminals from graph fragment + std::vector partsOfSpeech; + if (m_options.partsOfSpeechFactor) { + const Subgraph &graphFragment = rule.GetGraphFragment(); + graphFragment.GetPartsOfSpeech(partsOfSpeech); + } + + // Write the source side of the rule to sourceSS. + for (std::vector::const_iterator p(sourceRHS.begin()); + p != sourceRHS.end(); ++p) { + WriteSymbol(*p, sourceSS); + sourceSS << " "; + } + if (m_options.conditionOnTargetLhs) { + WriteSymbol(rule.GetTargetLHS(), sourceSS); + } else { + WriteSymbol(rule.GetSourceLHS(), sourceSS); + } + + // Write the target side of the rule to targetSS. + int targetTerminalIndex = 0; + for (std::vector::const_iterator p(targetRHS.begin()); + p != targetRHS.end(); ++p) { + WriteSymbol(*p, targetSS); + // If parts-of-speech as a factor requested: write part-of-speech + if (m_options.partsOfSpeechFactor && (p->GetType() != NonTerminal)) { + assert(targetTerminalIndex + +#include "Subgraph.h" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace GHKM +{ + +struct Options; +class ScfgRule; +class Symbol; + +class ScfgRuleWriter +{ +public: + ScfgRuleWriter(std::ostream &fwd, std::ostream &inv, const Options &options) + : m_fwd(fwd) + , m_inv(inv) + , m_options(options) {} + + void Write(const ScfgRule &rule, size_t lineNum, bool printEndl=true); + +private: + // Disallow copying + ScfgRuleWriter(const ScfgRuleWriter &); + ScfgRuleWriter &operator=(const ScfgRuleWriter &); + + void WriteStandardFormat(const ScfgRule &, std::ostream &, std::ostream &); + void WriteUnpairedFormat(const ScfgRule &, std::ostream &, std::ostream &); + void WriteSymbol(const Symbol &, std::ostream &); + + std::ostream &m_fwd; + std::ostream &m_inv; + const Options &m_options; +}; + +} // namespace GHKM +} // namespace Syntax +} // namespace MosesTraining diff --git a/mosesdecoder/phrase-extract/extract-ghkm/Span.cpp b/mosesdecoder/phrase-extract/extract-ghkm/Span.cpp new file mode 100644 index 0000000000000000000000000000000000000000..9524a843c2d21360a64776ee7069574c82f312af --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-ghkm/Span.cpp @@ -0,0 +1,51 @@ +/*********************************************************************** + Moses - statistical machine translation system + Copyright (C) 2006-2011 University of Edinburgh + + This library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + This library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with this library; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +***********************************************************************/ + +#include "Span.h" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace GHKM +{ + +bool SpansIntersect(const Span &a, const ContiguousSpan &b) +{ + for (Span::const_iterator p = a.begin(); p != a.end(); ++p) { + if (*p >= b.first && *p <= b.second) { + return true; + } + } + return false; +} + +ContiguousSpan Closure(const Span &s) +{ + ContiguousSpan result(-1,-1); + if (!s.empty()) { + result.first = *(s.begin()); + result.second = *(s.rbegin()); + } + return result; +} + +} // namespace GHKM +} // namespace Syntax +} // namespace MosesTraining diff --git a/mosesdecoder/phrase-extract/extract-ghkm/Span.h b/mosesdecoder/phrase-extract/extract-ghkm/Span.h new file mode 100644 index 0000000000000000000000000000000000000000..fa4be48f075eff60a69e863faaacebfbcf545395 --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-ghkm/Span.h @@ -0,0 +1,45 @@ +/*********************************************************************** + Moses - statistical machine translation system + Copyright (C) 2006-2011 University of Edinburgh + + This library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + This library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with this library; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +***********************************************************************/ + +#pragma once +#ifndef EXTRACT_GHKM_SPAN_H_ +#define EXTRACT_GHKM_SPAN_H_ + +#include +#include + +namespace MosesTraining +{ +namespace Syntax +{ +namespace GHKM +{ + +typedef std::set Span; +typedef std::pair ContiguousSpan; + +bool SpansIntersect(const Span &, const ContiguousSpan &); + +ContiguousSpan Closure(const Span &); + +} // namespace MosesTraining +} // namespace Syntax +} // namespace GHKM + +#endif diff --git a/mosesdecoder/phrase-extract/extract-ghkm/StsgRule.cpp b/mosesdecoder/phrase-extract/extract-ghkm/StsgRule.cpp new file mode 100644 index 0000000000000000000000000000000000000000..08b4b64ff20f2843ec5bf61483f1080b3c20a499 --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-ghkm/StsgRule.cpp @@ -0,0 +1,97 @@ +#include "StsgRule.h" + +#include + +#include "Node.h" +#include "Subgraph.h" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace GHKM +{ + +StsgRule::StsgRule(const Subgraph &fragment) + : m_targetSide(fragment, true) +{ + // Source side + + const std::set &sinkNodes = fragment.GetLeaves(); + + // Collect the subset of sink nodes that excludes target nodes with + // empty spans. + std::vector productiveSinks; + productiveSinks.reserve(sinkNodes.size()); + for (std::set::const_iterator p = sinkNodes.begin(); + p != sinkNodes.end(); ++p) { + const Node *sink = *p; + if (!sink->GetSpan().empty()) { + productiveSinks.push_back(sink); + } + } + + // Sort them into the order defined by their spans. + std::sort(productiveSinks.begin(), productiveSinks.end(), PartitionOrderComp); + + // Build a map from target nodes to source-order indices, so that we + // can construct the Alignment object later. + std::map > sinkToSourceIndices; + std::map nonTermSinkToSourceIndex; + + m_sourceSide.reserve(productiveSinks.size()); + int srcIndex = 0; + int nonTermCount = 0; + for (std::vector::const_iterator p = productiveSinks.begin(); + p != productiveSinks.end(); ++p, ++srcIndex) { + const Node &sink = **p; + if (sink.GetType() == TREE) { + m_sourceSide.push_back(Symbol("X", NonTerminal)); + sinkToSourceIndices[&sink].push_back(srcIndex); + nonTermSinkToSourceIndex[&sink] = nonTermCount++; + } else { + assert(sink.GetType() == SOURCE); + m_sourceSide.push_back(Symbol(sink.GetLabel(), Terminal)); + // Add all aligned target words to the sinkToSourceIndices map + const std::vector &parents(sink.GetParents()); + for (std::vector::const_iterator q = parents.begin(); + q != parents.end(); ++q) { + if ((*q)->GetType() == TARGET) { + sinkToSourceIndices[*q].push_back(srcIndex); + } + } + } + } + + // Alignment + + std::vector targetLeaves; + m_targetSide.GetTargetLeaves(targetLeaves); + + m_alignment.reserve(targetLeaves.size()); + m_nonTermAlignment.resize(nonTermCount); + + for (int i = 0, j = 0; i < targetLeaves.size(); ++i) { + const Node *leaf = targetLeaves[i]; + assert(leaf->GetType() != SOURCE); + if (leaf->GetSpan().empty()) { + continue; + } + std::map >::iterator p = + sinkToSourceIndices.find(leaf); + assert(p != sinkToSourceIndices.end()); + std::vector &sourceNodes = p->second; + for (std::vector::iterator r = sourceNodes.begin(); + r != sourceNodes.end(); ++r) { + int srcIndex = *r; + m_alignment.push_back(std::make_pair(srcIndex, i)); + } + if (leaf->GetType() == TREE) { + m_nonTermAlignment[nonTermSinkToSourceIndex[leaf]] = j++; + } + } +} + +} // namespace GHKM +} // namespace Syntax +} // namespace MosesTraining diff --git a/mosesdecoder/phrase-extract/extract-ghkm/StsgRule.h b/mosesdecoder/phrase-extract/extract-ghkm/StsgRule.h new file mode 100644 index 0000000000000000000000000000000000000000..a29cfb99ae954bb24928ee659041cd911d177123 --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-ghkm/StsgRule.h @@ -0,0 +1,47 @@ +#pragma once +#ifndef EXTRACT_GHKM_STSG_RULE_H_ +#define EXTRACT_GHKM_STSG_RULE_H_ + +#include + +#include "Rule.h" +#include "Subgraph.h" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace GHKM +{ + +class Node; + +class StsgRule : public Rule +{ +public: + StsgRule(const Subgraph &fragment); + + const std::vector &GetSourceSide() const { + return m_sourceSide; + } + const Subgraph &GetTargetSide() const { + return m_targetSide; + } + const std::vector &GetNonTermAlignment() const { + return m_nonTermAlignment; + } + int Scope() const { + return Rule::Scope(m_sourceSide); + } + +private: + std::vector m_sourceSide; + Subgraph m_targetSide; + std::vector m_nonTermAlignment; +}; + +} // namespace GHKM +} // namespace Syntax +} // namespace MosesTraining + +#endif diff --git a/mosesdecoder/phrase-extract/extract-ghkm/StsgRuleWriter.cpp b/mosesdecoder/phrase-extract/extract-ghkm/StsgRuleWriter.cpp new file mode 100644 index 0000000000000000000000000000000000000000..c1f0b7ad11762d9f3794564fa8dd2bd058421b25 --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-ghkm/StsgRuleWriter.cpp @@ -0,0 +1,98 @@ +#include "StsgRuleWriter.h" + +#include +#include +#include +#include +#include +#include + +#include "Alignment.h" +#include "Options.h" +#include "StsgRule.h" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace GHKM +{ + +void StsgRuleWriter::Write(const StsgRule &rule) +{ + std::ostringstream sourceSS; + std::ostringstream targetSS; + + // Write the source side of the rule to sourceSS. + const std::vector &sourceSide = rule.GetSourceSide(); + for (std::size_t i = 0; i < sourceSide.size(); ++i) { + const Symbol &symbol = sourceSide[i]; + if (i > 0) { + sourceSS << " "; + } + if (symbol.GetType() == NonTerminal) { + sourceSS << "[X]"; + } else { + sourceSS << symbol.GetValue(); + } + } + + // Write the target side of the rule to targetSS. + rule.GetTargetSide().PrintTree(targetSS); + + // Write the rule to the forward and inverse extract files. + if (m_options.t2s) { + // If model is tree-to-string then flip the source and target. + m_fwd << targetSS.str() << " ||| " << sourceSS.str() << " |||"; + m_inv << sourceSS.str() << " ||| " << targetSS.str() << " |||"; + } else { + m_fwd << sourceSS.str() << " ||| " << targetSS.str() << " |||"; + m_inv << targetSS.str() << " ||| " << sourceSS.str() << " |||"; + } + + // Write the non-terminal alignments. + const std::vector &nonTermAlignment = rule.GetNonTermAlignment(); + for (int srcIndex = 0; srcIndex < nonTermAlignment.size(); ++srcIndex) { + int tgtIndex = nonTermAlignment[srcIndex]; + if (m_options.t2s) { + // If model is tree-to-string then flip the source and target. + m_fwd << " " << tgtIndex << "-" << srcIndex; + m_inv << " " << srcIndex << "-" << tgtIndex; + } else { + m_fwd << " " << srcIndex << "-" << tgtIndex; + m_inv << " " << tgtIndex << "-" << srcIndex; + } + } + m_fwd << " |||"; + m_inv << " |||"; + + // Write the symbol alignments. + const Alignment &alignment = rule.GetAlignment(); + for (Alignment::const_iterator p = alignment.begin(); + p != alignment.end(); ++p) { + if (m_options.t2s) { + // If model is tree-to-string then flip the source and target. + m_fwd << " " << p->second << "-" << p->first; + m_inv << " " << p->first << "-" << p->second; + } else { + m_fwd << " " << p->first << "-" << p->second; + m_inv << " " << p->second << "-" << p->first; + } + } + + // Write a count of 1. + m_fwd << " ||| 1"; + m_inv << " ||| 1"; + + // Write the PCFG score (if requested). + if (m_options.pcfg) { + m_fwd << " ||| " << std::exp(rule.GetTargetSide().GetPcfgScore()); + } + + m_fwd << std::endl; + m_inv << std::endl; +} + +} // namespace GHKM +} // namespace Syntax +} // namespace MosesTraining diff --git a/mosesdecoder/phrase-extract/extract-ghkm/StsgRuleWriter.h b/mosesdecoder/phrase-extract/extract-ghkm/StsgRuleWriter.h new file mode 100644 index 0000000000000000000000000000000000000000..e7cbeb53db97068e064d5b07c5aed06834255a76 --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-ghkm/StsgRuleWriter.h @@ -0,0 +1,44 @@ +#pragma once +#ifndef EXTRACT_GHKM_STSG_RULE_WRITER_H_ +#define EXTRACT_GHKM_STSG_RULE_WRITER_H_ + +#include + +#include "Subgraph.h" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace GHKM +{ + +struct Options; +class StsgRule; +class Symbol; + +class StsgRuleWriter +{ +public: + StsgRuleWriter(std::ostream &fwd, std::ostream &inv, const Options &options) + : m_fwd(fwd) + , m_inv(inv) + , m_options(options) {} + + void Write(const StsgRule &rule); + +private: + // Disallow copying + StsgRuleWriter(const StsgRuleWriter &); + StsgRuleWriter &operator=(const StsgRuleWriter &); + + std::ostream &m_fwd; + std::ostream &m_inv; + const Options &m_options; +}; + +} // namespace GHKM +} // namespace Syntax +} // namespace MosesTraining + +#endif diff --git a/mosesdecoder/phrase-extract/extract-ghkm/Subgraph.cpp b/mosesdecoder/phrase-extract/extract-ghkm/Subgraph.cpp new file mode 100644 index 0000000000000000000000000000000000000000..e2e2f6c9930be17e254606427602c77be8925b4f --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-ghkm/Subgraph.cpp @@ -0,0 +1,206 @@ +/*********************************************************************** + Moses - statistical machine translation system + Copyright (C) 2006-2011 University of Edinburgh + + This library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + This library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with this library; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +***********************************************************************/ + +#include + +#include "Node.h" +#include "Subgraph.h" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace GHKM +{ + +void Subgraph::GetTargetLeaves(std::vector &result) const +{ + result.clear(); + GetTargetLeaves(m_root, result); +} + +void Subgraph::GetTargetLeaves(const Node *root, + std::vector &result) const +{ + if (root->GetType() == TARGET || m_leaves.find(root) != m_leaves.end()) { + result.push_back(root); + } else { + const std::vector &children = root->GetChildren(); + for (std::vector::const_iterator p(children.begin()); + p != children.end(); ++p) { + GetTargetLeaves(*p, result); + } + } +} + +int Subgraph::CountNodes(const Node *n) const +{ + if (n->GetType() != TREE) { + return 0; + } + if (IsTrivial()) { + return 1; + } + int count = 1; + const std::vector &children = n->GetChildren(); + for (std::vector::const_iterator p = children.begin(); + p != children.end(); ++p) { + const Node *child = *p; + if (m_leaves.find(child) == m_leaves.end()) { + count += CountNodes(child); + } else if (child->GetType() == TREE) { + ++count; + } + } + return count; +} + +int Subgraph::CalcSize(const Node *n) const +{ + if (n->GetType() != TREE || n->IsPreterminal()) { + return 0; + } + if (IsTrivial()) { + return 1; + } + int count = 1; + const std::vector &children = n->GetChildren(); + for (std::vector::const_iterator p = children.begin(); + p != children.end(); ++p) { + if (m_leaves.find(*p) == m_leaves.end()) { + count += CalcSize(*p); + } + } + return count; +} + +int Subgraph::CalcDepth(const Node *n) const +{ + if (n->GetType() != TREE || n->IsPreterminal() || m_leaves.empty()) { + return 0; + } + int maxChildDepth = 0; + const std::vector &children = n->GetChildren(); + for (std::vector::const_iterator p = children.begin(); + p != children.end(); ++p) { + if (m_leaves.find(*p) == m_leaves.end()) { + maxChildDepth = std::max(maxChildDepth, CalcDepth(*p)); + } + } + return maxChildDepth + 1; +} + +float Subgraph::CalcPcfgScore() const +{ + if (m_root->GetType() != TREE || m_leaves.empty()) { + return 0.0f; + } + float score = m_root->GetPcfgScore(); + for (std::set::const_iterator p = m_leaves.begin(); + p != m_leaves.end(); ++p) { + const Node *leaf = *p; + if (leaf->GetType() == TREE) { + score -= leaf->GetPcfgScore(); + } + } + return score; +} + +void Subgraph::PrintTree(std::ostream &out) const +{ + RecursivelyPrintTree(m_root,out); +} + +void Subgraph::RecursivelyPrintTree(const Node *n, std::ostream &out) const +{ + NodeType nodeType = n->GetType(); + if (nodeType == TREE) { + out << "[" << n->GetLabel(); + if (m_leaves.find(n) == m_leaves.end()) { + const std::vector &children = n->GetChildren(); + for (std::vector::const_iterator p(children.begin()); + p != children.end(); ++p) { + Node *child = *p; + if (child->GetType() == SOURCE) { + // This is possible due to the heuristic for attaching unaligned + // source words. + continue; + } + out << " "; + RecursivelyPrintTree(child,out); + } + } + out << "]"; + } else if (nodeType == TARGET) { + out << n->GetLabel(); + } +} + +void Subgraph::PrintPartsOfSpeech(std::ostream &out) const +{ + RecursivelyPrintPartsOfSpeech(m_root,out); +} + +void Subgraph::RecursivelyPrintPartsOfSpeech(const Node *n, std::ostream &out) const +{ + NodeType nodeType = n->GetType(); + if (nodeType == TREE) { + if (m_leaves.find(n) == m_leaves.end()) { + const std::vector &children = n->GetChildren(); + for (std::vector::const_iterator p(children.begin()); + p != children.end(); ++p) { + Node *child = *p; + if (child->GetType() == TARGET) { + out << " " << n->GetLabel(); + } else { + RecursivelyPrintPartsOfSpeech(child,out); + } + } + } + } +} + +void Subgraph::GetPartsOfSpeech(std::vector &out) const +{ + out.clear(); + RecursivelyGetPartsOfSpeech(m_root,out); +} + +void Subgraph::RecursivelyGetPartsOfSpeech(const Node *n, std::vector &out) const +{ + NodeType nodeType = n->GetType(); + if (nodeType == TREE) { + if (m_leaves.find(n) == m_leaves.end()) { + const std::vector &children = n->GetChildren(); + for (std::vector::const_iterator p(children.begin()); + p != children.end(); ++p) { + Node *child = *p; + if (child->GetType() == TARGET) { + out.push_back(n->GetLabel()); + } else { + RecursivelyGetPartsOfSpeech(child,out); + } + } + } + } +} + +} // namespace MosesTraining +} // namespace Syntax +} // namespace GHKM diff --git a/mosesdecoder/phrase-extract/extract-ghkm/Subgraph.h b/mosesdecoder/phrase-extract/extract-ghkm/Subgraph.h new file mode 100644 index 0000000000000000000000000000000000000000..1f1234421c6d0ede418abee7df21e11fdf4fe718 --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-ghkm/Subgraph.h @@ -0,0 +1,143 @@ +/*********************************************************************** + Moses - statistical machine translation system + Copyright (C) 2006-2011 University of Edinburgh + + This library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + This library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with this library; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +***********************************************************************/ + +#pragma once + +#include +#include + +#include "Node.h" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace GHKM +{ + +class Node; + +class Subgraph +{ +public: + Subgraph(const Node *root) + : m_root(root) + , m_depth(0) + , m_size(root->GetType() == TREE ? 1 : 0) + , m_nodeCount(1) + , m_pcfgScore(0.0f) {} + + Subgraph(const Node *root, const std::set &leaves) + : m_root(root) + , m_leaves(leaves) + , m_depth(-1) + , m_size(-1) + , m_nodeCount(-1) + , m_pcfgScore(0.0f) { + m_depth = CalcDepth(m_root); + m_size = CalcSize(m_root); + m_nodeCount = CountNodes(m_root); + m_pcfgScore = CalcPcfgScore(); + } + + Subgraph(const Subgraph &other, bool targetOnly=false) + : m_root(other.m_root) + , m_leaves(other.m_leaves) + , m_depth(other.m_depth) + , m_size(other.m_size) + , m_nodeCount(other.m_nodeCount) + , m_pcfgScore(other.m_pcfgScore) { + if (targetOnly && m_root->GetType() != SOURCE) { + // Replace any source-word sink nodes with their parents (except for + // the special case where the parent is a non-word tree node -- see + // below). + std::set targetLeaves; + for (std::set::const_iterator p = m_leaves.begin(); + p != m_leaves.end(); ++p) { + const Node *leaf = *p; + if (leaf->GetType() != SOURCE) { + targetLeaves.insert(leaf); + } else { + const std::vector &parents = leaf->GetParents(); + for (std::vector::const_iterator q = parents.begin(); + q != parents.end(); ++q) { + const Node *parent = *q; + // Only add parents that are words, not tree nodes since those + // are never sink nodes. (A source word can have a tree node as + // its parent due to the heuristic for handling unaligned source + // words). + if (parent->GetType() == TARGET) { + targetLeaves.insert(*q); + } + } + } + } + m_leaves.swap(targetLeaves); + } + } + + const Node *GetRoot() const { + return m_root; + } + const std::set &GetLeaves() const { + return m_leaves; + } + int GetDepth() const { + return m_depth; + } + int GetSize() const { + return m_size; + } + int GetNodeCount() const { + return m_nodeCount; + } + float GetPcfgScore() const { + return m_pcfgScore; + } + + bool IsTrivial() const { + return m_leaves.empty(); + } + + void GetTargetLeaves(std::vector &) const; + void PrintTree(std::ostream &out) const; + void PrintPartsOfSpeech(std::ostream &out) const; + void GetPartsOfSpeech(std::vector &out) const; + +private: + void GetTargetLeaves(const Node *, std::vector &) const; + int CalcDepth(const Node *) const; + int CalcSize(const Node *) const; + float CalcPcfgScore() const; + int CountNodes(const Node *) const; + void RecursivelyPrintTree(const Node *n, std::ostream &out) const; + void RecursivelyPrintPartsOfSpeech(const Node *n, std::ostream &out) const; + void RecursivelyGetPartsOfSpeech(const Node *n, std::vector &out) const; + + const Node *m_root; + std::set m_leaves; + int m_depth; + int m_size; + int m_nodeCount; + float m_pcfgScore; +}; + +} // namespace GHKM +} // namespace Syntax +} // namespace MosesTraining diff --git a/mosesdecoder/phrase-extract/extract-mixed-syntax/AlignedSentence.cpp b/mosesdecoder/phrase-extract/extract-mixed-syntax/AlignedSentence.cpp new file mode 100644 index 0000000000000000000000000000000000000000..b6b74454e9cb76ddb348776c9e4e755df89cc047 --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-mixed-syntax/AlignedSentence.cpp @@ -0,0 +1,195 @@ +/* + * AlignedSentence.cpp + * + * Created on: 18 Feb 2014 + * Author: s0565741 + */ + +#include +#include "moses/Util.h" +#include "AlignedSentence.h" +#include "Parameter.h" + +using namespace std; + + +///////////////////////////////////////////////////////////////////////////////// +AlignedSentence::AlignedSentence(int lineNum, + const std::string &source, + const std::string &target, + const std::string &alignment) + :m_lineNum(lineNum) +{ + PopulateWordVec(m_source, source); + PopulateWordVec(m_target, target); + PopulateAlignment(alignment); +} + +AlignedSentence::~AlignedSentence() +{ + Moses::RemoveAllInColl(m_source); + Moses::RemoveAllInColl(m_target); +} + +void AlignedSentence::PopulateWordVec(Phrase &vec, const std::string &line) +{ + std::vector toks; + Moses::Tokenize(toks, line); + + vec.resize(toks.size()); + for (size_t i = 0; i < vec.size(); ++i) { + const string &tok = toks[i]; + Word *word = new Word(i, tok); + vec[i] = word; + } +} + +void AlignedSentence::PopulateAlignment(const std::string &line) +{ + vector alignStr; + Moses::Tokenize(alignStr, line); + + for (size_t i = 0; i < alignStr.size(); ++i) { + vector alignPair; + Moses::Tokenize(alignPair, alignStr[i], "-"); + assert(alignPair.size() == 2); + + int sourcePos = alignPair[0]; + int targetPos = alignPair[1]; + + if (sourcePos >= m_source.size()) { + cerr << "ERROR1:AlignedSentence=" << Debug() << endl; + cerr << "m_source=" << m_source.size() << endl; + abort(); + } + assert(sourcePos < m_source.size()); + assert(targetPos < m_target.size()); + Word *sourceWord = m_source[sourcePos]; + Word *targetWord = m_target[targetPos]; + + sourceWord->AddAlignment(targetWord); + targetWord->AddAlignment(sourceWord); + } +} + +std::string AlignedSentence::Debug() const +{ + stringstream out; + out << "m_lineNum:"; + out << m_lineNum; + out << endl; + + out << "m_source:"; + out << m_source.Debug(); + out << endl; + + out << "m_target:"; + out << m_target.Debug(); + out << endl; + + out << "consistent phrases:" << endl; + out << m_consistentPhrases.Debug(); + out << endl; + + return out.str(); +} + +std::vector AlignedSentence::GetSourceAlignmentCount() const +{ + vector ret(m_source.size()); + + for (size_t i = 0; i < m_source.size(); ++i) { + const Word &word = *m_source[i]; + ret[i] = word.GetAlignmentIndex().size(); + } + return ret; +} + +void AlignedSentence::Create(const Parameter ¶ms) +{ + CreateConsistentPhrases(params); + m_consistentPhrases.AddHieroNonTerms(params); +} + +void AlignedSentence::CreateConsistentPhrases(const Parameter ¶ms) +{ + int countT = m_target.size(); + int countS = m_source.size(); + + m_consistentPhrases.Initialize(countS); + + // check alignments for target phrase startT...endT + for(int lengthT=1; + lengthT <= params.maxSpan && lengthT <= countT; + lengthT++) { + for(int startT=0; startT < countT-(lengthT-1); startT++) { + + // that's nice to have + int endT = startT + lengthT - 1; + + // find find aligned source words + // first: find minimum and maximum source word + int minS = 9999; + int maxS = -1; + vector< int > usedS = GetSourceAlignmentCount(); + for(int ti=startT; ti<=endT; ti++) { + const Word &word = *m_target[ti]; + const std::set &alignment = word.GetAlignmentIndex(); + + std::set::const_iterator iterAlign; + for(iterAlign = alignment.begin(); iterAlign != alignment.end(); ++iterAlign) { + int si = *iterAlign; + if (simaxS) { + maxS = si; + } + usedS[ si ]--; + } + } + + // unaligned phrases are not allowed + if( maxS == -1 ) + continue; + + // source phrase has to be within limits + size_t width = maxS - minS + 1; + + if( width < params.minSpan ) + continue; + + if( width > params.maxSpan ) + continue; + + // check if source words are aligned to out of bound target words + bool out_of_bounds = false; + for(int si=minS; si<=maxS && !out_of_bounds; si++) + if (usedS[si]>0) { + out_of_bounds = true; + } + + // if out of bound, you gotta go + if (out_of_bounds) + continue; + + // done with all the checks, lets go over all consistent phrase pairs + // start point of source phrase may retreat over unaligned + for(int startS=minS; + (startS>=0 && + startS>maxS - params.maxSpan && // within length limit + (startS==minS || m_source[startS]->GetAlignment().size()==0)); // unaligned + startS--) { + // end point of source phrase may advance over unaligned + for(int endS=maxS; + (endSGetAlignment().size()==0)); // unaligned + endS++) { + + // take note that this is a valid phrase alignment + m_consistentPhrases.Add(startS, endS, startT, endT, params); + } + } + } + } +} diff --git a/mosesdecoder/phrase-extract/extract-mixed-syntax/AlignedSentenceSyntax.h b/mosesdecoder/phrase-extract/extract-mixed-syntax/AlignedSentenceSyntax.h new file mode 100644 index 0000000000000000000000000000000000000000..db6764ceeba156dfba77c2bbcd60e6cceb467835 --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-mixed-syntax/AlignedSentenceSyntax.h @@ -0,0 +1,46 @@ +/* + * AlignedSentenceSyntax.h + * + * Created on: 26 Feb 2014 + * Author: hieu + */ + +#pragma once + +#include "AlignedSentence.h" +#include "SyntaxTree.h" +#include "pugixml.hpp" + +class AlignedSentenceSyntax : public AlignedSentence +{ +public: + AlignedSentenceSyntax(int lineNum, + const std::string &source, + const std::string &target, + const std::string &alignment); + virtual ~AlignedSentenceSyntax(); + + void Create(const Parameter ¶ms); + + //virtual std::string Debug() const; +protected: + std::string m_sourceStr, m_targetStr, m_alignmentStr; + SyntaxTree m_sourceTree, m_targetTree; + + void XMLParse(Phrase &output, + SyntaxTree &tree, + const std::string input, + const Parameter ¶ms); + void XMLParse(Phrase &output, + SyntaxTree &tree, + const pugi::xml_node &parentNode, + const Parameter ¶ms); + void CreateNonTerms(); + void CreateNonTerms(ConsistentPhrase &cp, + const SyntaxTree::Labels &sourceLabels, + const SyntaxTree::Labels &targetLabels); + void Populate(bool isSyntax, int mixedSyntaxType, const Parameter ¶ms, + std::string line, Phrase &phrase, SyntaxTree &tree); + +}; + diff --git a/mosesdecoder/phrase-extract/extract-mixed-syntax/ConsistentPhrase.cpp b/mosesdecoder/phrase-extract/extract-mixed-syntax/ConsistentPhrase.cpp new file mode 100644 index 0000000000000000000000000000000000000000..6ac22b1f7e16fa006e01ff13e4432c818a04b2ba --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-mixed-syntax/ConsistentPhrase.cpp @@ -0,0 +1,67 @@ +/* + * ConsistentPhrase.cpp + * + * Created on: 20 Feb 2014 + * Author: hieu + */ +#include +#include "ConsistentPhrase.h" +#include "Word.h" +#include "NonTerm.h" +#include "Parameter.h" + +using namespace std; + +ConsistentPhrase::ConsistentPhrase( + int sourceStart, int sourceEnd, + int targetStart, int targetEnd, + const Parameter ¶ms) + :corners(4) + ,m_hieroNonTerm(*this, params.hieroNonTerm, params.hieroNonTerm) +{ + corners[0] = sourceStart; + corners[1] = sourceEnd; + corners[2] = targetStart; + corners[3] = targetEnd; +} + +ConsistentPhrase::~ConsistentPhrase() +{ + // TODO Auto-generated destructor stub +} + +bool ConsistentPhrase::operator<(const ConsistentPhrase &other) const +{ + return corners < other.corners; +} + +void ConsistentPhrase::AddNonTerms(const std::string &source, + const std::string &target) +{ + m_nonTerms.push_back(NonTerm(*this, source, target)); +} + +bool ConsistentPhrase::TargetOverlap(const ConsistentPhrase &other) const +{ + if ( other.corners[3] < corners[2] || other.corners[2] > corners[3]) + return false; + + return true; +} + +std::string ConsistentPhrase::Debug() const +{ + stringstream out; + out << "[" << corners[0] << "-" << corners[1] + << "][" << corners[2] << "-" << corners[3] << "]"; + + out << "NT:"; + for (size_t i = 0; i < m_nonTerms.size(); ++i) { + const NonTerm &nonTerm = m_nonTerms[i]; + out << nonTerm.GetLabel(Moses::Input) << ":" << nonTerm.GetLabel(Moses::Output); + } + + return out.str(); +} + + diff --git a/mosesdecoder/phrase-extract/extract-mixed-syntax/ConsistentPhrases.cpp b/mosesdecoder/phrase-extract/extract-mixed-syntax/ConsistentPhrases.cpp new file mode 100644 index 0000000000000000000000000000000000000000..b1d64fc5468c273709b4eecddf5057fd86c40ae7 --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-mixed-syntax/ConsistentPhrases.cpp @@ -0,0 +1,104 @@ +/* + * ConsistentPhrases.cpp + * + * Created on: 20 Feb 2014 + * Author: hieu + */ +#include +#include +#include "ConsistentPhrases.h" +#include "NonTerm.h" +#include "Parameter.h" +#include "moses/Util.h" + +using namespace std; + +ConsistentPhrases::ConsistentPhrases() +{ +} + +ConsistentPhrases::~ConsistentPhrases() +{ + for (size_t start = 0; start < m_coll.size(); ++start) { + std::vector &allSourceStart = m_coll[start]; + + for (size_t size = 0; size < allSourceStart.size(); ++size) { + Coll &coll = allSourceStart[size]; + Moses::RemoveAllInColl(coll); + } + } +} + +void ConsistentPhrases::Initialize(size_t size) +{ + m_coll.resize(size); + + for (size_t sourceStart = 0; sourceStart < size; ++sourceStart) { + std::vector &allSourceStart = m_coll[sourceStart]; + allSourceStart.resize(size - sourceStart); + } +} + +void ConsistentPhrases::Add(int sourceStart, int sourceEnd, + int targetStart, int targetEnd, + const Parameter ¶ms) +{ + Coll &coll = m_coll[sourceStart][sourceEnd - sourceStart]; + ConsistentPhrase *cp = new ConsistentPhrase(sourceStart, sourceEnd, + targetStart, targetEnd, + params); + + assert(coll.find(cp) == coll.end()); + coll.insert(cp); +} + +const ConsistentPhrases::Coll &ConsistentPhrases::GetColl(int sourceStart, int sourceEnd) const +{ + const std::vector &allSourceStart = m_coll[sourceStart]; + const Coll &ret = allSourceStart[sourceEnd - sourceStart]; + return ret; +} + +ConsistentPhrases::Coll &ConsistentPhrases::GetColl(int sourceStart, int sourceEnd) +{ + std::vector &allSourceStart = m_coll[sourceStart]; + Coll &ret = allSourceStart[sourceEnd - sourceStart]; + return ret; +} + +std::string ConsistentPhrases::Debug() const +{ + std::stringstream out; + for (size_t start = 0; start < m_coll.size(); ++start) { + const std::vector &allSourceStart = m_coll[start]; + + for (size_t size = 0; size < allSourceStart.size(); ++size) { + const Coll &coll = allSourceStart[size]; + + Coll::const_iterator iter; + for (iter = coll.begin(); iter != coll.end(); ++iter) { + const ConsistentPhrase &consistentPhrase = **iter; + out << consistentPhrase.Debug() << endl; + } + } + } + + return out.str(); +} + +void ConsistentPhrases::AddHieroNonTerms(const Parameter ¶ms) +{ + // add [X] labels everywhere + for (size_t i = 0; i < m_coll.size(); ++i) { + vector &inner = m_coll[i]; + for (size_t j = 0; j < inner.size(); ++j) { + ConsistentPhrases::Coll &coll = inner[j]; + ConsistentPhrases::Coll::iterator iter; + for (iter = coll.begin(); iter != coll.end(); ++iter) { + ConsistentPhrase &cp = **iter; + cp.AddNonTerms(params.hieroNonTerm, params.hieroNonTerm); + } + } + } +} + diff --git a/mosesdecoder/phrase-extract/extract-mixed-syntax/Jamfile b/mosesdecoder/phrase-extract/extract-mixed-syntax/Jamfile new file mode 100644 index 0000000000000000000000000000000000000000..520cd65cbb10e336b4909727ab1a7162c8689fd5 --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-mixed-syntax/Jamfile @@ -0,0 +1,2 @@ +exe extract-mixed-syntax : Main.cpp AlignedSentence.cpp AlignedSentenceSyntax.cpp ConsistentPhrase.cpp ConsistentPhrases.cpp NonTerm.cpp Parameter.cpp Phrase.cpp pugixml.cpp Rule.cpp RulePhrase.cpp Rules.cpp RuleSymbol.cpp SyntaxTree.cpp Word.cpp ..//deps ../..//z ../..//boost_iostreams ../..//boost_program_options ../../moses//moses : .. ; + diff --git a/mosesdecoder/phrase-extract/extract-mixed-syntax/Parameter.cpp b/mosesdecoder/phrase-extract/extract-mixed-syntax/Parameter.cpp new file mode 100644 index 0000000000000000000000000000000000000000..a9d4f8ea0740bb1b0ed928c69a02fe66e1ba7540 --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-mixed-syntax/Parameter.cpp @@ -0,0 +1,74 @@ +/* + * Parameter.cpp + * + * Created on: 17 Feb 2014 + * Author: hieu + */ +#include "Parameter.h" +#include "moses/Util.h" +#include "util/exception.hh" + +using namespace std; + +Parameter::Parameter() + :maxSpan(10) + ,minSpan(0) + ,maxNonTerm(2) + ,maxHieroNonTerm(999) + ,maxSymbolsTarget(999) + ,maxSymbolsSource(5) + ,minHoleSource(2) + ,minHoleSourceSyntax(1) + ,sentenceOffset(0) + ,nonTermConsecSource(false) + ,requireAlignedWord(true) + ,fractionalCounting(true) + ,gzOutput(false) + + ,hieroNonTerm("[X]") + ,sourceSyntax(false) + ,targetSyntax(false) + + ,mixedSyntaxType(0) + ,multiLabel(0) + ,nonTermConsecSourceMixed(true) + ,hieroSourceLHS(false) + ,maxSpanFreeNonTermSource(0) + ,nieceTerminal(true) + ,maxScope(UNDEFINED) + ,minScope(0) + + ,spanLength(false) + ,ruleLength(false) + ,nonTermContext(false) + ,nonTermContextTarget(false) + ,nonTermContextFactor(0) + + ,numSourceFactors(1) + ,numTargetFactors(1) + + ,nonTermConsecSourceMixedSyntax(1) +{} + +Parameter::~Parameter() +{ + // TODO Auto-generated destructor stub +} + +void Parameter::SetScopeSpan(const std::string &str) +{ + scopeSpanStr = str; + vector toks1; + Moses::Tokenize(toks1, str, ":"); + + for (size_t i = 0; i < toks1.size(); ++i) { + const string &tok1 = toks1[i]; + + vector toks2; + Moses::Tokenize(toks2, tok1, ","); + UTIL_THROW_IF2(toks2.size() != 2, "Format is min,max:min,max... String is " << tok1); + + std::pair values(toks2[0], toks2[1]); + scopeSpan.push_back(values); + } +} diff --git a/mosesdecoder/phrase-extract/extract-mixed-syntax/Parameter.h b/mosesdecoder/phrase-extract/extract-mixed-syntax/Parameter.h new file mode 100644 index 0000000000000000000000000000000000000000..bf56be4a1bb000a9a5284efa756ea95b8756a55e --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-mixed-syntax/Parameter.h @@ -0,0 +1,66 @@ +/* + * Parameter.h + * + * Created on: 17 Feb 2014 + * Author: hieu + */ +#pragma once + +#include +#include +#include + +#define UNDEFINED std::numeric_limits::max() + +class Parameter +{ +public: + Parameter(); + virtual ~Parameter(); + + int maxSpan; + int minSpan; + int maxNonTerm; + int maxHieroNonTerm; + int maxSymbolsTarget; + int maxSymbolsSource; + int minHoleSource; + int minHoleSourceSyntax; + + long sentenceOffset; + + bool nonTermConsecSource; + bool requireAlignedWord; + bool fractionalCounting; + bool gzOutput; + + std::string hieroNonTerm; + std::string gluePath; + + bool sourceSyntax, targetSyntax; + + int mixedSyntaxType, multiLabel; + bool nonTermConsecSourceMixed; + bool hieroSourceLHS; + int maxSpanFreeNonTermSource; + bool nieceTerminal; + int maxScope, minScope; + + // properties + bool spanLength; + bool ruleLength; + bool nonTermContext; + bool nonTermContextTarget; + int nonTermContextFactor; + + int numSourceFactors, numTargetFactors; + + int nonTermConsecSourceMixedSyntax; + + std::string scopeSpanStr; + std::vector > scopeSpan; + + void SetScopeSpan(const std::string &str); + +}; + diff --git a/mosesdecoder/phrase-extract/extract-mixed-syntax/Phrase.cpp b/mosesdecoder/phrase-extract/extract-mixed-syntax/Phrase.cpp new file mode 100644 index 0000000000000000000000000000000000000000..613c67a2630e3bab73b4e69b731cf0344e22a3e0 --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-mixed-syntax/Phrase.cpp @@ -0,0 +1,14 @@ +#include +#include "Phrase.h" + +std::string Phrase::Debug() const +{ + std::stringstream out; + + for (size_t i = 0; i < size(); ++i) { + Word &word = *at(i); + out << word.Debug() << " "; + } + + return out.str(); +} diff --git a/mosesdecoder/phrase-extract/extract-mixed-syntax/RulePhrase.cpp b/mosesdecoder/phrase-extract/extract-mixed-syntax/RulePhrase.cpp new file mode 100644 index 0000000000000000000000000000000000000000..1894ade7d9636febff18f9aea121c92dfce84197 --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-mixed-syntax/RulePhrase.cpp @@ -0,0 +1,50 @@ +/* + * RulePhrase.cpp + * + * Created on: 26 Feb 2014 + * Author: hieu + */ + +#include +#include "RulePhrase.h" +#include "RuleSymbol.h" + +using namespace std; + +extern bool g_debug; + +int RulePhrase::Compare(const RulePhrase &other) const +{ + if (GetSize() != other.GetSize()) { + return GetSize() < other.GetSize() ? -1 : +1; + } + + for (size_t i = 0; i < m_coll.size(); ++i) { + const RuleSymbol &symbol = *m_coll[i]; + const RuleSymbol &otherSymbol = *other.m_coll[i]; + int compare = symbol.Compare(otherSymbol); + + if (compare) { + return compare; + } + } + + return 0; +} + +void RulePhrase::Output(std::ostream &out) const +{ + for (size_t i = 0; i < m_coll.size(); ++i) { + const RuleSymbol &symbol = *m_coll[i]; + symbol.Output(out); + out << " "; + } +} + +std::string RulePhrase::Debug() const +{ + std::stringstream out; + Output(out); + return out.str(); +} + diff --git a/mosesdecoder/phrase-extract/extract-mixed-syntax/RulePhrase.h b/mosesdecoder/phrase-extract/extract-mixed-syntax/RulePhrase.h new file mode 100644 index 0000000000000000000000000000000000000000..0527293ba3f92b2ef56d26ad213a4246d33f6975 --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-mixed-syntax/RulePhrase.h @@ -0,0 +1,49 @@ +/* + * RulePhrase.h + * + * Created on: 26 Feb 2014 + * Author: hieu + */ + +#ifndef RULEPHRASE_H_ +#define RULEPHRASE_H_ + +#include +#include +#include + +class RuleSymbol; + +// a phrase of terms and non-terms for 1 side of a rule +class RulePhrase +{ +public: + typedef std::vector Coll; + Coll m_coll; + + size_t GetSize() const { + return m_coll.size(); + } + + void Add(const RuleSymbol *symbol) { + m_coll.push_back(symbol); + } + + const RuleSymbol* operator[](size_t index) const { + return m_coll[index]; + } + + const RuleSymbol* Front() const { + return m_coll.front(); + } + const RuleSymbol* Back() const { + return m_coll.back(); + } + + int Compare(const RulePhrase &other) const; + + void Output(std::ostream &out) const; + std::string Debug() const; +}; + +#endif /* RULEPHRASE_H_ */ diff --git a/mosesdecoder/phrase-extract/extract-mixed-syntax/RuleSymbol.h b/mosesdecoder/phrase-extract/extract-mixed-syntax/RuleSymbol.h new file mode 100644 index 0000000000000000000000000000000000000000..a9909664d26f601f0acc4fd98a9335572d1627a1 --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-mixed-syntax/RuleSymbol.h @@ -0,0 +1,32 @@ +/* + * RuleSymbol.h + * + * Created on: 21 Feb 2014 + * Author: hieu + */ + +#ifndef RULESYMBOL_H_ +#define RULESYMBOL_H_ + +#include +#include + +// base class - terminal or non-term +class RuleSymbol +{ +public: + RuleSymbol(); + virtual ~RuleSymbol(); + + virtual bool IsNonTerm() const = 0; + + virtual std::string Debug() const = 0; + virtual void Output(std::ostream &out) const = 0; + + virtual std::string GetString() const = 0; + + int Compare(const RuleSymbol &other) const; + +}; + +#endif /* RULESYMBOL_H_ */ diff --git a/mosesdecoder/phrase-extract/extract-mixed-syntax/Rules.cpp b/mosesdecoder/phrase-extract/extract-mixed-syntax/Rules.cpp new file mode 100644 index 0000000000000000000000000000000000000000..740afb172a307901689bd3f8f6d301c02e4adf78 --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-mixed-syntax/Rules.cpp @@ -0,0 +1,225 @@ +/* + * Rules.cpp + * + * Created on: 20 Feb 2014 + * Author: hieu + */ + +#include +#include "Rules.h" +#include "ConsistentPhrase.h" +#include "ConsistentPhrases.h" +#include "AlignedSentence.h" +#include "Rule.h" +#include "Parameter.h" +#include "moses/Util.h" + +using namespace std; + +extern bool g_debug; + +Rules::Rules(const AlignedSentence &alignedSentence) + :m_alignedSentence(alignedSentence) +{ +} + +Rules::~Rules() +{ + Moses::RemoveAllInColl(m_keepRules); +} + +void Rules::CreateRules(const ConsistentPhrase &cp, + const Parameter ¶ms) +{ + if (params.hieroSourceLHS) { + const NonTerm &nonTerm = cp.GetHieroNonTerm(); + CreateRule(nonTerm, params); + } else { + const ConsistentPhrase::NonTerms &nonTerms = cp.GetNonTerms(); + for (size_t i = 0; i < nonTerms.size(); ++i) { + const NonTerm &nonTerm = nonTerms[i]; + CreateRule(nonTerm, params); + } + } +} + +void Rules::CreateRule(const NonTerm &nonTerm, + const Parameter ¶ms) +{ + Rule *rule = new Rule(nonTerm, m_alignedSentence); + + rule->Prevalidate(params); + rule->CreateTarget(params); + rule->CreateProperties(params); + + if (rule->CanRecurse()) { + Extend(*rule, params); + } + + if (rule->IsValid()) { + m_keepRules.insert(rule); + } else { + delete rule; + } + +} + +void Rules::Extend(const Parameter ¶ms) +{ + const ConsistentPhrases &allCPS = m_alignedSentence.GetConsistentPhrases(); + + size_t size = m_alignedSentence.GetPhrase(Moses::Input).size(); + for (size_t sourceStart = 0; sourceStart < size; ++sourceStart) { + for (size_t sourceEnd = sourceStart; sourceEnd < size; ++sourceEnd) { + const ConsistentPhrases::Coll &cps = allCPS.GetColl(sourceStart, sourceEnd); + + ConsistentPhrases::Coll::const_iterator iter; + for (iter = cps.begin(); iter != cps.end(); ++iter) { + const ConsistentPhrase &cp = **iter; + CreateRules(cp, params); + } + } + } +} + +void Rules::Extend(const Rule &rule, const Parameter ¶ms) +{ + const ConsistentPhrases &allCPS = m_alignedSentence.GetConsistentPhrases(); + int sourceMin = rule.GetNextSourcePosForNonTerm(); + + int ruleStart = rule.GetConsistentPhrase().corners[0]; + int ruleEnd = rule.GetConsistentPhrase().corners[1]; + + for (int sourceStart = sourceMin; sourceStart <= ruleEnd; ++sourceStart) { + for (int sourceEnd = sourceStart; sourceEnd <= ruleEnd; ++sourceEnd) { + if (sourceStart == ruleStart && sourceEnd == ruleEnd) { + // don't cover whole rule with 1 non-term + continue; + } + + const ConsistentPhrases::Coll &cps = allCPS.GetColl(sourceStart, sourceEnd); + Extend(rule, cps, params); + } + } +} + +void Rules::Extend(const Rule &rule, const ConsistentPhrases::Coll &cps, const Parameter ¶ms) +{ + ConsistentPhrases::Coll::const_iterator iter; + for (iter = cps.begin(); iter != cps.end(); ++iter) { + const ConsistentPhrase &cp = **iter; + Extend(rule, cp, params); + } +} + +void Rules::Extend(const Rule &rule, const ConsistentPhrase &cp, const Parameter ¶ms) +{ + const ConsistentPhrase::NonTerms &nonTerms = cp.GetNonTerms(); + for (size_t i = 0; i < nonTerms.size(); ++i) { + const NonTerm &nonTerm = nonTerms[i]; + + Rule *newRule = new Rule(rule, nonTerm); + newRule->Prevalidate(params); + newRule->CreateTarget(params); + newRule->CreateProperties(params); + + if (newRule->CanRecurse()) { + // recursively extend + Extend(*newRule, params); + } + + if (newRule->IsValid()) { + m_keepRules.insert(newRule); + } else { + delete newRule; + } + } +} + +std::string Rules::Debug() const +{ + stringstream out; + + std::set::const_iterator iter; + out << "m_keepRules:" << endl; + for (iter = m_keepRules.begin(); iter != m_keepRules.end(); ++iter) { + const Rule &rule = **iter; + out << rule.Debug() << endl; + } + + return out.str(); +} + +void Rules::Output(std::ostream &out, bool forward, const Parameter ¶ms) const +{ + std::set::const_iterator iter; + for (iter = m_mergeRules.begin(); iter != m_mergeRules.end(); ++iter) { + const Rule &rule = **iter; + rule.Output(out, forward); + out << endl; + } +} + +void Rules::Consolidate(const Parameter ¶ms) +{ + if (params.fractionalCounting) { + CalcFractionalCount(); + } else { + std::set::iterator iter; + for (iter = m_keepRules.begin(); iter != m_keepRules.end(); ++iter) { + Rule &rule = **iter; + rule.SetCount(1); + } + } + + MergeRules(params); +} + +void Rules::MergeRules(const Parameter ¶ms) +{ + typedef std::set MergeRules; + + std::set::const_iterator iterOrig; + for (iterOrig = m_keepRules.begin(); iterOrig != m_keepRules.end(); ++iterOrig) { + Rule *origRule = *iterOrig; + + pair inserted = m_mergeRules.insert(origRule); + if (!inserted.second) { + // already there, just add count + Rule &rule = **inserted.first; + float newCount = rule.GetCount() + origRule->GetCount(); + rule.SetCount(newCount); + } + } +} + +void Rules::CalcFractionalCount() +{ + typedef std::set RuleColl; + typedef std::map RuleByConsistentPhrase; + RuleByConsistentPhrase allRules; + + // sort by source AND target ranges + std::set::const_iterator iter; + for (iter = m_keepRules.begin(); iter != m_keepRules.end(); ++iter) { + Rule *rule = *iter; + const ConsistentPhrase &cp = rule->GetConsistentPhrase(); + RuleColl &ruleColl = allRules[&cp]; + ruleColl.insert(rule); + } + + // fractional count + RuleByConsistentPhrase::iterator iterOuter; + for (iterOuter = allRules.begin(); iterOuter != allRules.end(); ++iterOuter) { + RuleColl &rules = iterOuter->second; + + RuleColl::iterator iterInner; + for (iterInner = rules.begin(); iterInner != rules.end(); ++iterInner) { + Rule &rule = **iterInner; + rule.SetCount(1.0f / (float) rules.size()); + } + } + +} + + diff --git a/mosesdecoder/phrase-extract/extract-mixed-syntax/Rules.h b/mosesdecoder/phrase-extract/extract-mixed-syntax/Rules.h new file mode 100644 index 0000000000000000000000000000000000000000..a45416af7b5b08fe3ffe04e5b8a5c483b979e444 --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-mixed-syntax/Rules.h @@ -0,0 +1,76 @@ +/* + * Rules.h + * + * Created on: 20 Feb 2014 + * Author: hieu + */ + +#pragma once + +#include +#include +#include "ConsistentPhrases.h" +#include "Rule.h" + +extern bool g_debug; + +class AlignedSentence; +class Parameter; + +struct CompareRules { + bool operator()(const Rule *a, const Rule *b) { + int compare; + + compare = a->GetPhrase(Moses::Input).Compare(b->GetPhrase(Moses::Input)); + if (compare) return compare < 0; + + compare = a->GetPhrase(Moses::Output).Compare(b->GetPhrase(Moses::Output)); + if (compare) return compare < 0; + + if (a->GetAlignments() != b->GetAlignments()) { + return a->GetAlignments() < b->GetAlignments(); + } + + if (a->GetLHS().GetString() != b->GetLHS().GetString()) { + return a->GetLHS().GetString() < b->GetLHS().GetString(); + } + + if (a->GetProperties() != b->GetProperties()) { + return a->GetProperties() < b->GetProperties(); + } + + return false; + } +}; + +class Rules +{ +public: + Rules(const AlignedSentence &alignedSentence); + virtual ~Rules(); + void Extend(const Parameter ¶ms); + void Consolidate(const Parameter ¶ms); + + std::string Debug() const; + void Output(std::ostream &out, bool forward, const Parameter ¶ms) const; + +protected: + const AlignedSentence &m_alignedSentence; + std::set m_keepRules; + std::set m_mergeRules; + + void Extend(const Rule &rule, const Parameter ¶ms); + void Extend(const Rule &rule, const ConsistentPhrases::Coll &cps, const Parameter ¶ms); + void Extend(const Rule &rule, const ConsistentPhrase &cp, const Parameter ¶ms); + + // create original rules + void CreateRules(const ConsistentPhrase &cp, + const Parameter ¶ms); + void CreateRule(const NonTerm &nonTerm, + const Parameter ¶ms); + + void MergeRules(const Parameter ¶ms); + void CalcFractionalCount(); + +}; + diff --git a/mosesdecoder/phrase-extract/extract-mixed-syntax/SyntaxTree.cpp b/mosesdecoder/phrase-extract/extract-mixed-syntax/SyntaxTree.cpp new file mode 100644 index 0000000000000000000000000000000000000000..b96c661672ae80e745fce46aec0611140d4fa602 --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-mixed-syntax/SyntaxTree.cpp @@ -0,0 +1,46 @@ +#include +#include +#include "SyntaxTree.h" +#include "Parameter.h" + +using namespace std; + +void SyntaxTree::Add(int startPos, int endPos, const std::string &label, const Parameter ¶ms) +{ + //cerr << "add " << label << " to " << "[" << startPos << "-" << endPos << "]" << endl; + + Range range(startPos, endPos); + Labels &labels = m_coll[range]; + + bool add = true; + if (labels.size()) { + if (params.multiLabel == 1) { + // delete the label in collection and add new + assert(labels.size() == 1); + labels.clear(); + } else if (params.multiLabel == 2) { + // ignore this label + add = false; + } + } + + if (add) { + labels.push_back(label); + } +} + +void SyntaxTree::AddToAll(const std::string &label) +{ + Coll::iterator iter; + for (iter = m_coll.begin(); iter != m_coll.end(); ++iter) { + Labels &labels = iter->second; + labels.push_back(label); + } +} + +const SyntaxTree::Labels &SyntaxTree::Find(int startPos, int endPos) const +{ + Coll::const_iterator iter; + iter = m_coll.find(Range(startPos, endPos)); + return (iter == m_coll.end()) ? m_defaultLabels : iter->second; +} diff --git a/mosesdecoder/phrase-extract/extract-mixed-syntax/Word.h b/mosesdecoder/phrase-extract/extract-mixed-syntax/Word.h new file mode 100644 index 0000000000000000000000000000000000000000..80ee20ba924b13f3618bfc934706292c0c9b4434 --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-mixed-syntax/Word.h @@ -0,0 +1,53 @@ +/* + * Word.h + * + * Created on: 18 Feb 2014 + * Author: s0565741 + */ +#pragma once + +#include +#include +#include "RuleSymbol.h" + +// a terminal +class Word : public RuleSymbol +{ +public: + Word(const Word&); // do not implement + Word(int pos, const std::string &str); + virtual ~Word(); + + virtual bool IsNonTerm() const { + return false; + } + + std::string GetString() const { + return m_str; + } + + std::string GetString(int factor) const; + + int GetPos() const { + return m_pos; + } + + void AddAlignment(const Word *other); + + const std::set &GetAlignment() const { + return m_alignment; + } + + std::set GetAlignmentIndex() const; + + void Output(std::ostream &out) const; + std::string Debug() const; + + int CompareString(const Word &other) const; + +protected: + int m_pos; // original position in sentence, NOT in lattice + std::string m_str; + std::set m_alignment; +}; + diff --git a/mosesdecoder/phrase-extract/extract-mixed-syntax/gzfilebuf.h b/mosesdecoder/phrase-extract/extract-mixed-syntax/gzfilebuf.h new file mode 100644 index 0000000000000000000000000000000000000000..e070da306ccad733bdfd988321174e156e74e246 --- /dev/null +++ b/mosesdecoder/phrase-extract/extract-mixed-syntax/gzfilebuf.h @@ -0,0 +1,88 @@ +#ifndef moses_gzfile_buf_h +#define moses_gzfile_buf_h + +#include +#include +#include +#include + +class gzfilebuf : public std::streambuf +{ +public: + gzfilebuf(const char *filename) { + _gzf = gzopen(filename, "rb"); + if (!_gzf) + throw std::runtime_error("Could not open " + std::string(filename) + "."); + setg (_buff+sizeof(int), // beginning of putback area + _buff+sizeof(int), // read position + _buff+sizeof(int)); // end position + } + ~gzfilebuf() { + gzclose(_gzf); + } +protected: + virtual int_type overflow (int_type c) { + throw; + } + + // write multiple characters + virtual + std::streamsize xsputn (const char* s, + std::streamsize num) { + throw; + } + + virtual std::streampos seekpos ( std::streampos sp, std::ios_base::openmode which = std::ios_base::in | std::ios_base::out ) { + throw; + } + + //read one character + virtual int_type underflow () { + // is read position before end of _buff? + if (gptr() < egptr()) { + return traits_type::to_int_type(*gptr()); + } + + /* process size of putback area + * - use number of characters read + * - but at most four + */ + unsigned int numPutback = gptr() - eback(); + if (numPutback > sizeof(int)) { + numPutback = sizeof(int); + } + + /* copy up to four characters previously read into + * the putback _buff (area of first four characters) + */ + std::memmove (_buff+(sizeof(int)-numPutback), gptr()-numPutback, + numPutback); + + // read new characters + int num = gzread(_gzf, _buff+sizeof(int), _buffsize-sizeof(int)); + if (num <= 0) { + // ERROR or EOF + return EOF; + } + + // reset _buff pointers + setg (_buff+(sizeof(int)-numPutback), // beginning of putback area + _buff+sizeof(int), // read position + _buff+sizeof(int)+num); // end of buffer + + // return next character + return traits_type::to_int_type(*gptr()); + } + + std::streamsize xsgetn (char* s, + std::streamsize num) { + return gzread(_gzf,s,num); + } + +private: + gzFile _gzf; + static const unsigned int _buffsize = 1024; + char _buff[_buffsize]; +}; + +#endif diff --git a/mosesdecoder/phrase-extract/filter-rule-table/FilterRuleTable.cpp b/mosesdecoder/phrase-extract/filter-rule-table/FilterRuleTable.cpp new file mode 100644 index 0000000000000000000000000000000000000000..24c2803a7c5b18f797b3cde25a54e92c15afbffe --- /dev/null +++ b/mosesdecoder/phrase-extract/filter-rule-table/FilterRuleTable.cpp @@ -0,0 +1,231 @@ +#include "FilterRuleTable.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include "syntax-common/exception.h" +#include "syntax-common/xml_tree_parser.h" + +#include "InputFileStream.h" + +#include "ForestTsgFilter.h" +#include "Options.h" +#include "StringCfgFilter.h" +#include "StringForest.h" +#include "StringForestParser.h" +#include "TreeCfgFilter.h" +#include "TreeTsgFilter.h" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace FilterRuleTable +{ + +int FilterRuleTable::Main(int argc, char *argv[]) +{ + enum TestSentenceFormat { + kUnknownTestSentenceFormat, + kString, + kTree, + kForest + }; + + enum SourceSideRuleFormat { + kUnknownSourceSideRuleFormat, + kCfg, + kTsg + }; + + // Process command-line options. + Options options; + ProcessOptions(argc, argv, options); + + // Open input file. + Moses::InputFileStream testStream(options.testSetFile); + + // Determine the expected test sentence format and source-side rule format + // based on the argument to the options.model parameter. + TestSentenceFormat testSentenceFormat = kUnknownTestSentenceFormat; + SourceSideRuleFormat sourceSideRuleFormat = kUnknownSourceSideRuleFormat; + if (options.model == "hierarchical" || options.model == "s2t") { + testSentenceFormat = kString; + sourceSideRuleFormat = kCfg; + } else if (options.model == "t2s") { + testSentenceFormat = kTree; + sourceSideRuleFormat = kTsg; + } else if (options.model == "t2s-scfg") { + testSentenceFormat = kTree; + sourceSideRuleFormat = kCfg; + } else if (options.model == "f2s") { + testSentenceFormat = kForest; + sourceSideRuleFormat = kTsg; + } else { + Error(std::string("unsupported model type: ") + options.model); + } + + // Read the test sentences then set up and run the filter. + if (testSentenceFormat == kString) { + assert(sourceSideRuleFormat == kCfg); + std::vector > testStrings; + ReadTestSet(testStream, testStrings); + StringCfgFilter filter(testStrings); + filter.Filter(std::cin, std::cout); + } else if (testSentenceFormat == kTree) { + std::vector > testTrees; + ReadTestSet(testStream, testTrees); + if (sourceSideRuleFormat == kCfg) { + // TODO Implement TreeCfgFilter + Warn("tree/cfg filtering algorithm not implemented: input will be copied unchanged to output"); + TreeCfgFilter filter(testTrees); + filter.Filter(std::cin, std::cout); + } else if (sourceSideRuleFormat == kTsg) { + TreeTsgFilter filter(testTrees); + filter.Filter(std::cin, std::cout); + } else { + assert(false); + } + } else if (testSentenceFormat == kForest) { + std::vector > testForests; + ReadTestSet(testStream, testForests); + assert(sourceSideRuleFormat == kTsg); + ForestTsgFilter filter(testForests); + filter.Filter(std::cin, std::cout); + } + + return 0; +} + +void FilterRuleTable::ReadTestSet( + std::istream &input, + std::vector > &sentences) +{ + int lineNum = 0; + std::string line; + while (std::getline(input, line)) { + ++lineNum; + if (line.empty()) { + std::cerr << "skipping blank test sentence at line " << lineNum + << std::endl; + continue; + } + sentences.push_back(boost::make_shared(line)); + } +} + +void FilterRuleTable::ReadTestSet( + std::istream &input, std::vector > &sentences) +{ + XmlTreeParser parser; + int lineNum = 0; + std::string line; + while (std::getline(input, line)) { + ++lineNum; + if (line.empty()) { + std::cerr << "skipping blank test sentence at line " << lineNum + << std::endl; + continue; + } + sentences.push_back( + boost::shared_ptr(parser.Parse(line).release())); + } +} + +void FilterRuleTable::ReadTestSet( + std::istream &input, + std::vector > &sentences) +{ + StringForestParser end; + int sentNum = 0; + for (StringForestParser p(input); p != end; ++p) { + ++sentNum; + if (p->forest->vertices.empty()) { + std::cerr << "skipping sentence " << sentNum << ": forest is empty" + << std::endl; + continue; + } + sentences.push_back(p->forest); + } +} + +void FilterRuleTable::ProcessOptions(int argc, char *argv[], + Options &options) const +{ + namespace po = boost::program_options; + namespace cls = boost::program_options::command_line_style; + + // Construct the 'top' of the usage message: the bit that comes before the + // options list. + std::ostringstream usageTop; + usageTop << "Usage: " << name() + << " [OPTION]... MODEL TEST\n\n" + << "Filter for SCFG/STSG rule tables.\n\n" + << "Options"; + + // Construct the 'bottom' of the usage message. + std::ostringstream usageBottom; + usageBottom << "\nGiven a rule table on standard input and a set of test sentences, filters out\nthe rules that cannot be applied to any of the test sentences and writes the\nfiltered table to standard output. MODEL specifies the type of syntax model.\nThe following values are supported:\n\n" + << " hierarchical, s2t, t2s, t2s-scfg, f2s\n"; + + // Declare the command line options that are visible to the user. + po::options_description visible(usageTop.str()); + + // Declare the command line options that are hidden from the user + // (these are used as positional options). + po::options_description hidden("Hidden options"); + hidden.add_options() + ("Model", + po::value(&options.model), + "one of: hierarchical, s2t, t2s, t2s-scfg, f2s") + ("TestSetFile", + po::value(&options.testSetFile), + "test set file") + ; + + // Compose the full set of command-line options. + po::options_description cmdLineOptions; + cmdLineOptions.add(visible).add(hidden); + + // Register the positional options. + po::positional_options_description p; + p.add("Model", 1); + p.add("TestSetFile", 1); + + // Process the command-line. + po::variables_map vm; + try { + po::store(po::command_line_parser(argc, argv).style(MosesOptionStyle()). + options(cmdLineOptions).positional(p).run(), vm); + po::notify(vm); + } catch (const std::exception &e) { + std::ostringstream msg; + msg << e.what() << "\n\n" << visible << usageBottom.str(); + Error(msg.str()); + } + + if (vm.count("help")) { + std::cout << visible << usageBottom.str() << std::endl; + std::exit(0); + } + + // Check all positional options were given. + if (!vm.count("TestSetFile")) { + std::ostringstream msg; + std::cerr << visible << usageBottom.str() << std::endl; + std::exit(1); + } +} + +} // namespace FilterRuleTable +} // namespace Syntax +} // namespace MosesTraining diff --git a/mosesdecoder/phrase-extract/filter-rule-table/Main.cpp b/mosesdecoder/phrase-extract/filter-rule-table/Main.cpp new file mode 100644 index 0000000000000000000000000000000000000000..1a7e25016cb5a65bdc0252f12bef453bc673bfbc --- /dev/null +++ b/mosesdecoder/phrase-extract/filter-rule-table/Main.cpp @@ -0,0 +1,7 @@ +#include "FilterRuleTable.h" + +int main(int argc, char *argv[]) +{ + MosesTraining::Syntax::FilterRuleTable::FilterRuleTable tool; + return tool.Main(argc, argv); +} diff --git a/mosesdecoder/phrase-extract/filter-rule-table/Options.h b/mosesdecoder/phrase-extract/filter-rule-table/Options.h new file mode 100644 index 0000000000000000000000000000000000000000..c3871075ab0ecd3edf7f322d77b2fab5f324eb21 --- /dev/null +++ b/mosesdecoder/phrase-extract/filter-rule-table/Options.h @@ -0,0 +1,23 @@ +#pragma once + +#include + +namespace MosesTraining +{ +namespace Syntax +{ +namespace FilterRuleTable +{ + +struct Options { +public: + Options() {} + + // Positional options + std::string model; + std::string testSetFile; +}; + +} // namespace FilterRuleTable +} // namespace Syntax +} // namespace Moses diff --git a/mosesdecoder/phrase-extract/filter-rule-table/StringForestParser.cpp b/mosesdecoder/phrase-extract/filter-rule-table/StringForestParser.cpp new file mode 100644 index 0000000000000000000000000000000000000000..5fa6ca9b413b1a63df859c95e39eb21a3413a771 --- /dev/null +++ b/mosesdecoder/phrase-extract/filter-rule-table/StringForestParser.cpp @@ -0,0 +1,151 @@ +#include "StringForestParser.h" + +#include +#include + +#include + +#include "util/tokenize_piece.hh" + +#include "syntax-common/exception.h" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace FilterRuleTable +{ + +StringForestParser::StringForestParser() + : m_input(0) +{ +} + +StringForestParser::StringForestParser(std::istream &input) + : m_input(&input) +{ + ++(*this); +} + +StringForestParser &StringForestParser::operator++() +{ + if (!m_input) { + return *this; + } + m_vertexSet.clear(); + m_entry.forest.reset(new StringForest()); + if (!std::getline(*m_input, m_tmpLine)) { + m_input = 0; + return *this; + } + // The first line contains the sentence number. + ParseSentenceNumLine(m_tmpLine, m_entry.sentNum); + // The second line contains the sentence string. + std::getline(*m_input, m_entry.sentence); + // Subsequent lines contain hyperedges -- or a blank line if there was a + // parse failure -- terminated by a blank line. + std::getline(*m_input, m_tmpLine); + if (m_tmpLine == "") { // Parse failure + std::getline(*m_input, m_tmpLine); + assert(m_tmpLine == ""); + return *this; + } + while (m_tmpLine != "") { + ParseHyperedgeLine(m_tmpLine, *m_entry.forest); + std::getline(*m_input, m_tmpLine); + } + return *this; +} + +StringForest::Vertex *StringForestParser::AddOrDeleteVertex( + StringForest::Vertex *v) +{ + std::pair ret = m_vertexSet.insert(v); + if (ret.second) { + m_entry.forest->vertices.push_back(*ret.first); + } else { + delete v; + } + return *ret.first; +} + +void StringForestParser::ParseSentenceNumLine(const std::string &line, + std::size_t &sentNum) +{ + const util::AnyCharacter delimiter(" \t"); + util::TokenIter p(line, delimiter); + if (*p != "sentence") { + // FIXME + throw Exception(""); + } + ++p; + std::string tmp; + p->CopyToString(&tmp); + sentNum = std::atoi(tmp.c_str()); +} + +void StringForestParser::ParseHyperedgeLine(const std::string &line, + StringForest &forest) +{ + const util::AnyCharacter delimiter(" \t"); + util::TokenIter p(line, delimiter); + StringForest::Vertex *v = AddOrDeleteVertex(ParseVertex(*p)); + StringForest::Hyperedge *e = new StringForest::Hyperedge(); + e->head = v; + ++p; + if (*p != "=>") { + // FIXME + throw Exception(""); + } + for (++p; *p != "|||"; ++p) { + v = ParseVertex(*p); + if (v->value.start == -1) { + // Egret does not give start/end for terminals. + v->value.start = v->value.end = e->head->value.start; + } + e->tail.push_back(AddOrDeleteVertex(v)); + } + // Weight is ignored + e->head->incoming.push_back(e); +} + +StringForest::Vertex *StringForestParser::ParseVertex(const StringPiece &s) +{ + StringForest::Vertex *v = new StringForest::Vertex(); + std::size_t pos = s.rfind('['); + if (pos == std::string::npos) { + s.CopyToString(&v->value.symbol); + //v.value.symbol.isNonTerminal = false; + v->value.start = v->value.end = -1; + return v; + } + if (pos > 2 && s[pos-2] == '^' && s[pos-1] == 'g') { + s.substr(0, pos-2).CopyToString(&v->value.symbol); + } else { + s.substr(0, pos).CopyToString(&v->value.symbol); + } + //v.symbol.isNonTerminal = true; + std::size_t begin = pos + 1; + pos = s.find(',', begin+1); + std::string tmp; + s.substr(begin, pos-begin).CopyToString(&tmp); + v->value.start = std::atoi(tmp.c_str()); + s.substr(pos+1, s.size()-pos-2).CopyToString(&tmp); + v->value.end = std::atoi(tmp.c_str()); + return v; +} + +bool operator==(const StringForestParser &lhs, const StringForestParser &rhs) +{ + // TODO Is this right? Compare values of istreams if non-zero? + return lhs.m_input == rhs.m_input; +} + +bool operator!=(const StringForestParser &lhs, const StringForestParser &rhs) +{ + return !(lhs == rhs); +} + +} // namespace FilterRuleTable +} // namespace Syntax +} // namespace MosesTraining diff --git a/mosesdecoder/phrase-extract/filter-rule-table/StringForestParser.h b/mosesdecoder/phrase-extract/filter-rule-table/StringForestParser.h new file mode 100644 index 0000000000000000000000000000000000000000..8d4c2092ee7b4356d82e75f2c362b7459c9f832d --- /dev/null +++ b/mosesdecoder/phrase-extract/filter-rule-table/StringForestParser.h @@ -0,0 +1,88 @@ +#pragma once + +#include +#include +#include +#include + +#include +#include + +#include "util/string_piece.hh" + +#include "StringForest.h" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace FilterRuleTable +{ + +class StringForestParser +{ +public: + struct Entry { + std::size_t sentNum; + std::string sentence; + boost::shared_ptr forest; + }; + + StringForestParser(); + StringForestParser(std::istream &); + + Entry &operator*() { + return m_entry; + } + Entry *operator->() { + return &m_entry; + } + + StringForestParser &operator++(); + + friend bool operator==(const StringForestParser &, + const StringForestParser &); + friend bool operator!=(const StringForestParser &, + const StringForestParser &); + +private: + struct VertexSetHash { + std::size_t operator()(const StringForest::Vertex *v) const { + std::size_t seed = 0; + boost::hash_combine(seed, v->value.symbol); + boost::hash_combine(seed, v->value.start); + boost::hash_combine(seed, v->value.end); + return seed; + } + }; + + struct VertexSetPred { + bool operator()(const StringForest::Vertex *v, + const StringForest::Vertex *w) const { + return v->value.symbol == w->value.symbol && + v->value.start == w->value.start && + v->value.end == w->value.end; + } + }; + + typedef boost::unordered_set VertexSet; + + // Copying is not allowed + StringForestParser(const StringForestParser &); + StringForestParser &operator=(const StringForestParser &); + + StringForest::Vertex *AddOrDeleteVertex(StringForest::Vertex *); + void ParseHyperedgeLine(const std::string &, StringForest &); + void ParseSentenceNumLine(const std::string &, std::size_t &); + StringForest::Vertex *ParseVertex(const StringPiece &); + + Entry m_entry; + std::istream *m_input; + std::string m_tmpLine; + VertexSet m_vertexSet; +}; + +} // namespace FilterRuleTable +} // namespace Syntax +} // namespace MosesTraining diff --git a/mosesdecoder/phrase-extract/filter-rule-table/TreeCfgFilter.cpp b/mosesdecoder/phrase-extract/filter-rule-table/TreeCfgFilter.cpp new file mode 100644 index 0000000000000000000000000000000000000000..dc938ac1946b05915f23b9997fa3defbce549ec5 --- /dev/null +++ b/mosesdecoder/phrase-extract/filter-rule-table/TreeCfgFilter.cpp @@ -0,0 +1,30 @@ +#include "TreeCfgFilter.h" + +#include + +#include "util/string_piece_hash.hh" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace FilterRuleTable +{ + +TreeCfgFilter::TreeCfgFilter( + const std::vector > &sentences) +{ +} + +void TreeCfgFilter::Filter(std::istream &in, std::ostream &out) +{ + // TODO Implement filtering! + std::string line; + while (std::getline(in, line)) { + out << line << std::endl; + } +} + +} // namespace FilterRuleTable +} // namespace Syntax +} // namespace MosesTraining diff --git a/mosesdecoder/phrase-extract/filter-rule-table/TreeCfgFilter.h b/mosesdecoder/phrase-extract/filter-rule-table/TreeCfgFilter.h new file mode 100644 index 0000000000000000000000000000000000000000..3434ff2001cf5ee61241626c96a9473883adb7a0 --- /dev/null +++ b/mosesdecoder/phrase-extract/filter-rule-table/TreeCfgFilter.h @@ -0,0 +1,40 @@ +#pragma once + +#include +#include +#include +#include + +#include +#include + +#include "SyntaxTree.h" + +#include "syntax-common/numbered_set.h" +#include "syntax-common/tree.h" +#include "syntax-common/tree_fragment_tokenizer.h" + +#include "CfgFilter.h" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace FilterRuleTable +{ + +// Filters a rule table, discarding rules that cannot be applied to a given +// test set. The rule table must have a TSG source-side and the test sentences +// must be parse trees. +class TreeCfgFilter : public CfgFilter +{ +public: + // Initialize the filter for a given set of test sentences. + TreeCfgFilter(const std::vector > &); + + void Filter(std::istream &in, std::ostream &out); +}; + +} // namespace FilterRuleTable +} // namespace Syntax +} // namespace MosesTraining diff --git a/mosesdecoder/phrase-extract/filter-rule-table/TreeTsgFilter.cpp b/mosesdecoder/phrase-extract/filter-rule-table/TreeTsgFilter.cpp new file mode 100644 index 0000000000000000000000000000000000000000..b9c58228d77c757e4ef1c67127781af9b8126207 --- /dev/null +++ b/mosesdecoder/phrase-extract/filter-rule-table/TreeTsgFilter.cpp @@ -0,0 +1,120 @@ +#include "TreeTsgFilter.h" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace FilterRuleTable +{ + +TreeTsgFilter::TreeTsgFilter( + const std::vector > &sentences) +{ + // Convert each SyntaxTree to an IdTree. + m_sentences.reserve(sentences.size()); + for (std::vector >::const_iterator p = + sentences.begin(); p != sentences.end(); ++p) { + m_sentences.push_back(boost::shared_ptr(SyntaxTreeToIdTree(**p))); + } + + m_labelToTree.resize(m_testVocab.Size()); + // Construct a map from vocabulary Ids to IdTree nodes. + for (std::vector >::const_iterator p = + m_sentences.begin(); p != m_sentences.end(); ++p) { + AddNodesToMap(**p); + } +} + +TreeTsgFilter::IdTree *TreeTsgFilter::SyntaxTreeToIdTree(const SyntaxTree &s) +{ + IdTree *t = new IdTree(m_testVocab.Insert(s.value().label)); + const std::vector &sChildren = s.children(); + std::vector &tChildren = t->children(); + tChildren.reserve(sChildren.size()); + for (std::vector::const_iterator p = sChildren.begin(); + p != sChildren.end(); ++p) { + IdTree *child = SyntaxTreeToIdTree(**p); + child->parent() = t; + tChildren.push_back(child); + } + return t; +} + +void TreeTsgFilter::AddNodesToMap(const IdTree &tree) +{ + m_labelToTree[tree.value()].push_back(&tree); + const std::vector &children = tree.children(); + for (std::vector::const_iterator p = children.begin(); + p != children.end(); ++p) { + AddNodesToMap(**p); + } +} + +bool TreeTsgFilter::MatchFragment(const IdTree &fragment, + const std::vector &leaves) +{ + typedef std::vector TreeVec; + + // Determine which of the fragment's leaves has the smallest number of + // subtree matches in the test set. If the fragment contains a rare word + // (which is pretty likely assuming a Zipfian distribution) then we only + // have to try matching the fragment against a small number of potential + // match sites. + const IdTree *rarestLeaf = leaves[0]; + std::size_t lowestCount = m_labelToTree[rarestLeaf->value()].size(); + for (std::size_t i = 1; i < leaves.size(); ++i) { + const IdTree *leaf = leaves[i]; + std::size_t count = m_labelToTree[leaf->value()].size(); + if (count < lowestCount) { + lowestCount = count; + rarestLeaf = leaf; + } + } + + // Determine the depth of the chosen leaf. + const std::size_t depth = rarestLeaf->Depth(); + + // Try to match the rule fragment against the test set subtrees where a + // leaf match was found. + TreeVec &nodes = m_labelToTree[rarestLeaf->value()]; + for (TreeVec::const_iterator p = nodes.begin(); p != nodes.end(); ++p) { + // Navigate 'depth' positions up the subtree to find the root of the + // potential match site. + const IdTree *t = *p; + std::size_t d = depth; + while (d && t->parent()) { + t = t->parent(); + --d; + } + if (d > 0) { + // The potential match site is not tall enough. + continue; + } + if (MatchFragment(fragment, *t)) { + return true; + } + } + return false; +} + +bool TreeTsgFilter::MatchFragment(const IdTree &fragment, const IdTree &tree) +{ + if (fragment.value() != tree.value()) { + return false; + } + const std::vector &fragChildren = fragment.children(); + const std::vector &treeChildren = tree.children(); + if (!fragChildren.empty() && fragChildren.size() != treeChildren.size()) { + return false; + } + for (std::size_t i = 0; i < fragChildren.size(); ++i) { + if (!MatchFragment(*fragChildren[i], *treeChildren[i])) { + return false; + } + } + return true; +} + +} // namespace FilterRuleTable +} // namespace Syntax +} // namespace MosesTraining diff --git a/mosesdecoder/phrase-extract/filter-rule-table/TsgFilter.cpp b/mosesdecoder/phrase-extract/filter-rule-table/TsgFilter.cpp new file mode 100644 index 0000000000000000000000000000000000000000..e18c7d7f784669587891e0e56377c052a8ae4340 --- /dev/null +++ b/mosesdecoder/phrase-extract/filter-rule-table/TsgFilter.cpp @@ -0,0 +1,168 @@ +#include "TsgFilter.h" + +#include "boost/scoped_ptr.hpp" + +#include "util/string_piece.hh" +#include "util/string_piece_hash.hh" +#include "util/tokenize_piece.hh" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace FilterRuleTable +{ + +// Read a rule table from 'in' and filter it according to the test sentences. +// +// This involves testing TSG fragments for matches against at potential match +// sites in the set of test parse trees / forests. There are a few +// optimizations that make this reasonably fast in practice: +// +// Optimization 1 +// If a rule has the same TSG fragment as the previous rule then re-use the +// result of the previous filtering decision. +// +// Optimization 2 +// Test if the TSG fragment contains any symbols that don't occur in the +// symbol vocabulary of the test set. If it does then the rule can be +// discarded. +// +// Optimization 3 +// Prior to filtering, a map is constructed from each distinct test set tree / +// forest vertex symbol to the set of vertices having that symbol. During +// filtering, for each rule's TSG fragment the leaf with the smallest number of +// corresponding test nodes nodes is determined. Matching is only attempted +// at those sites (this is done in MatchFragment, which has tree- and +// forest-specific implementations). +// +// Some statistics from real data (WMT14, English-German, tree-version): +// +// 4.4M Parallel sentences (source-side parsed with Berkeley parser) +// 2.7K Test sentences (newstest2014) +// +// 73.4M Original rule table size (number of distinct, composed GHKM rules) +// 22.9M Number of rules with same source-side as previous rule +// 50.5M Number of rules requiring vocabulary matching test +// 24.1M Number of rules requiring full tree matching test +// 6.7M Number of rules retained after filtering +// +void TsgFilter::Filter(std::istream &in, std::ostream &out) +{ + const util::MultiCharacter delimiter("|||"); + + std::string line; + std::string prevLine; + StringPiece source; + bool keep = true; + int lineNum = 0; + std::vector tokens; + std::vector leaves; + + while (std::getline(in, line)) { + ++lineNum; + + // Read the source-side of the rule. + util::TokenIter it(line, delimiter); + + // Check if this rule has the same source-side as the previous rule. If + // it does then we already know whether or not to keep the rule. This + // optimisation is based on the assumption that the rule table is sorted + // (which is the case in the standard Moses training pipeline). + if (*it == source) { + if (keep) { + out << line << std::endl; + } + continue; + } + + // The source-side is different from the previous rule's. + source = *it; + + // Tokenize the source-side tree fragment. + tokens.clear(); + for (TreeFragmentTokenizer p(source); p != TreeFragmentTokenizer(); ++p) { + tokens.push_back(*p); + } + + // Construct an IdTree representing the source-side tree fragment. This + // will fail if the fragment contains any symbols that don't occur in + // m_testVocab and in that case the rule can be discarded. In practice, + // this catches a lot of discardable rules (see comment at the top of this + // function). If the fragment is successfully created then we attempt to + // match the tree fragment against the test trees. This test is exact, but + // slow. + int i = 0; + leaves.clear(); + boost::scoped_ptr fragment(BuildTree(tokens, i, leaves)); + keep = fragment.get() && MatchFragment(*fragment, leaves); + if (keep) { + out << line << std::endl; + } + + // Retain line for the next iteration (in order that the source StringPiece + // remains valid). + prevLine.swap(line); + } +} + +TsgFilter::IdTree *TsgFilter::BuildTree( + const std::vector &tokens, int &i, + std::vector &leaves) +{ + // The subtree starting at tokens[i] is either: + // 1. a single non-variable symbol (like NP or dog), or + // 2. a variable symbol (like [NP]), or + // 3. a subtree with children (like [NP [DT] [NN dog]]) + + // First check for case 1. + if (tokens[i].type == TreeFragmentToken_WORD) { + Vocabulary::IdType id = m_testVocab.Lookup(tokens[i++].value, + StringPieceCompatibleHash(), + StringPieceCompatibleEquals()); + if (id == Vocabulary::NullId()) { + return 0; + } + leaves.push_back(new IdTree(id)); + return leaves.back(); + } + + // We must be dealing with either case 2 or 3. Case 2 looks like case 3 but + // without the children. + assert(tokens[i].type == TreeFragmentToken_LSB); + + // Skip over the opening [ + ++i; + + // Read the root symbol of the subtree. + Vocabulary::IdType id = m_testVocab.Lookup(tokens[i++].value, + StringPieceCompatibleHash(), + StringPieceCompatibleEquals()); + if (id == Vocabulary::NullId()) { + return 0; + } + IdTree *root = new IdTree(id); + + // Read the children (in case 2 there won't be any). + while (tokens[i].type != TreeFragmentToken_RSB) { + IdTree *child = BuildTree(tokens, i, leaves); + if (!child) { + delete root; + return 0; + } + root->children().push_back(child); + child->parent() = root; + } + + if (root->IsLeaf()) { + leaves.push_back(root); + } + + // Skip over the closing ] and we're done. + ++i; + return root; +} + +} // namespace FilterRuleTable +} // namespace Syntax +} // namespace MosesTraining diff --git a/mosesdecoder/phrase-extract/pcfg-extract/main.cc b/mosesdecoder/phrase-extract/pcfg-extract/main.cc new file mode 100644 index 0000000000000000000000000000000000000000..010c04b00638061819fdfd30b44d53de86a9abd0 --- /dev/null +++ b/mosesdecoder/phrase-extract/pcfg-extract/main.cc @@ -0,0 +1,26 @@ +/*********************************************************************** + Moses - statistical machine translation system + Copyright (C) 2006-2012 University of Edinburgh + + This library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + This library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with this library; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +***********************************************************************/ + +#include "pcfg_extract.h" + +int main(int argc, char *argv[]) +{ + MosesTraining::Syntax::PCFG::PcfgExtract tool; + return tool.Main(argc, argv); +} diff --git a/mosesdecoder/phrase-extract/pcfg-extract/rule_collection.cc b/mosesdecoder/phrase-extract/pcfg-extract/rule_collection.cc new file mode 100644 index 0000000000000000000000000000000000000000..a814f82d675a5db7e11ba2c9353356515dfe9070 --- /dev/null +++ b/mosesdecoder/phrase-extract/pcfg-extract/rule_collection.cc @@ -0,0 +1,65 @@ +/*********************************************************************** + Moses - statistical machine translation system + Copyright (C) 2006-2012 University of Edinburgh + + This library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + This library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with this library; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +***********************************************************************/ + +#include "rule_collection.h" + +#include "syntax-common/pcfg.h" + +#include + +namespace MosesTraining +{ +namespace Syntax +{ +namespace PCFG +{ + +void RuleCollection::Add(std::size_t lhs, const std::vector &rhs) +{ + ++collection_[lhs][rhs]; +} + +void RuleCollection::CreatePcfg(Pcfg &pcfg) +{ + std::vector key; + for (const_iterator p = begin(); p != end(); ++p) { + std::size_t lhs = p->first; + const RhsCountMap &rhs_counts = p->second; + std::size_t total = 0; + for (RhsCountMap::const_iterator q = rhs_counts.begin(); + q != rhs_counts.end(); ++q) { + total += q->second; + } + for (RhsCountMap::const_iterator q = rhs_counts.begin(); + q != rhs_counts.end(); ++q) { + const std::vector &rhs = q->first; + std::size_t count = q->second; + double score = std::log(static_cast(count) / + static_cast(total)); + key.clear(); + key.push_back(lhs); + key.insert(key.end(), rhs.begin(), rhs.end()); + pcfg.Add(key, score); + } + } +} + +} // namespace PCFG +} // namespace Syntax +} // namespace MosesTraining diff --git a/mosesdecoder/phrase-extract/pcfg-extract/rule_extractor.cc b/mosesdecoder/phrase-extract/pcfg-extract/rule_extractor.cc new file mode 100644 index 0000000000000000000000000000000000000000..f20f2d9785d43a6cc5f492bad8f7b6ee49f27ab4 --- /dev/null +++ b/mosesdecoder/phrase-extract/pcfg-extract/rule_extractor.cc @@ -0,0 +1,56 @@ +/*********************************************************************** + Moses - statistical machine translation system + Copyright (C) 2006-2012 University of Edinburgh + + This library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + This library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with this library; if not, write to the Free Software + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA +***********************************************************************/ + +#include "rule_extractor.h" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace PCFG +{ + +RuleExtractor::RuleExtractor(Vocabulary &non_term_vocab) + : non_term_vocab_(non_term_vocab) +{ +} + +void RuleExtractor::Extract(const SyntaxTree &tree, RuleCollection &rc) const +{ + if (tree.IsLeaf() || tree.children()[0]->IsLeaf()) { + return; + } + + std::size_t lhs = non_term_vocab_.Insert(tree.value().label); + std::vector rhs; + + const std::vector &children = tree.children(); + rhs.reserve(children.size()); + for (std::vector::const_iterator p(children.begin()); + p != children.end(); ++p) { + const SyntaxTree &child = **p; + rhs.push_back(non_term_vocab_.Insert(child.value().label)); + Extract(child, rc); + } + rc.Add(lhs, rhs); +} + +} // namespace PCFG +} // namespace Syntax +} // namespace MosesTraining diff --git a/mosesdecoder/phrase-extract/score-stsg/LexicalTable.cpp b/mosesdecoder/phrase-extract/score-stsg/LexicalTable.cpp new file mode 100644 index 0000000000000000000000000000000000000000..48815ba26e46c17075423e1e4f1dcce8f592b4cf --- /dev/null +++ b/mosesdecoder/phrase-extract/score-stsg/LexicalTable.cpp @@ -0,0 +1,56 @@ +#include "LexicalTable.h" + +#include "util/tokenize_piece.hh" + +#include +#include + +namespace MosesTraining +{ +namespace Syntax +{ +namespace ScoreStsg +{ + +LexicalTable::LexicalTable(Vocabulary &srcVocab, Vocabulary &tgtVocab) + : m_srcVocab(srcVocab) + , m_tgtVocab(tgtVocab) +{ +} + +void LexicalTable::Load(std::istream &input) +{ + const util::AnyCharacter delimiter(" \t"); + + std::string line; + std::string tmp; + int i = 0; + while (getline(input, line)) { + ++i; + if (i%100000 == 0) { + std::cerr << "."; + } + + util::TokenIter it(line, delimiter); + + // Target word + it->CopyToString(&tmp); + Vocabulary::IdType tgtId = m_tgtVocab.Insert(tmp); + ++it; + + // Source word. + it->CopyToString(&tmp); + Vocabulary::IdType srcId = m_srcVocab.Insert(tmp); + ++it; + + // Probability. + it->CopyToString(&tmp); + double prob = atof(tmp.c_str()); + m_table[srcId][tgtId] = prob; + } + std::cerr << std::endl; +} + +} // namespace ScoreStsg +} // namespace Syntax +} // namespace MosesTraining diff --git a/mosesdecoder/phrase-extract/score-stsg/Main.cpp b/mosesdecoder/phrase-extract/score-stsg/Main.cpp new file mode 100644 index 0000000000000000000000000000000000000000..4a8f7a57fdd74379ecc36c7cc67a1ccaae0d92f3 --- /dev/null +++ b/mosesdecoder/phrase-extract/score-stsg/Main.cpp @@ -0,0 +1,7 @@ +#include "ScoreStsg.h" + +int main(int argc, char *argv[]) +{ + MosesTraining::Syntax::ScoreStsg::ScoreStsg tool; + return tool.Main(argc, argv); +} diff --git a/mosesdecoder/phrase-extract/score-stsg/Options.h b/mosesdecoder/phrase-extract/score-stsg/Options.h new file mode 100644 index 0000000000000000000000000000000000000000..25e63a5c0b92282bd65ebb986dfce78a0826167d --- /dev/null +++ b/mosesdecoder/phrase-extract/score-stsg/Options.h @@ -0,0 +1,44 @@ +#pragma once + +#include + +namespace MosesTraining +{ +namespace Syntax +{ +namespace ScoreStsg +{ + +struct Options { +public: + Options() + : goodTuring(false) + , inverse(false) + , kneserNey(false) + , logProb(false) + , minCountHierarchical(0) + , negLogProb(false) + , noLex(false) + , noWordAlignment(false) + , treeScore(false) {} + + // Positional options + std::string extractFile; + std::string lexFile; + std::string tableFile; + + // All other options + bool goodTuring; + bool inverse; + bool kneserNey; + bool logProb; + int minCountHierarchical; + bool negLogProb; + bool noLex; + bool noWordAlignment; + bool treeScore; +}; + +} // namespace ScoreStsg +} // namespace Syntax +} // namespace MosesTraining diff --git a/mosesdecoder/phrase-extract/score-stsg/RuleGroup.h b/mosesdecoder/phrase-extract/score-stsg/RuleGroup.h new file mode 100644 index 0000000000000000000000000000000000000000..1d03eeeb1c7951747b3549465dc9db42497f486d --- /dev/null +++ b/mosesdecoder/phrase-extract/score-stsg/RuleGroup.h @@ -0,0 +1,82 @@ +#pragma once + +#include +#include +#include + +#include "util/string_piece.hh" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace ScoreStsg +{ + +// A group of rules that share the same source-side. Rules are added through +// calls to SetNewSource() and AddRule(). They can then be accessed via the +// iterators. +// +// It is assumed that rules with the same (target, ntAlign, alignment) value +// will be added consecutively, and so will rules with the same +// (target, ntAlign) value. In other words, it is assumed that rules will be +// added in the order they occur in a correctly-sorted extract file. +class RuleGroup +{ +public: + // Stores the target-side and NT-alignment of a distinct rule. Also records + // the rule's count, the observed symbol alignments (plus their frequencies), + // and the tree score. + struct DistinctRule { + std::string target; + std::string ntAlign; + std::vector > alignments; + int count; + double treeScore; + }; + + typedef std::vector::const_iterator ConstIterator; + + // Begin and End iterators for iterating over the group's distinct rules. + ConstIterator Begin() const { + return m_distinctRules.begin(); + } + ConstIterator End() const { + return m_distinctRules.end(); + } + + // Get the current source-side value. + const std::string &GetSource() const { + return m_source; + } + + // Get the number of distinct rules. + int GetSize() const { + return m_distinctRules.size(); + } + + // Get the total count. + int GetTotalCount() const { + return m_totalCount; + } + + // Clear the rule group and set a new source-side value. This must be + // done once for every new source-side value, prior to the first call to + // AddRule(). + void SetNewSource(const StringPiece &source); + + // Add a rule. To determine rule distinctness, the target and ntAlign + // values will be checked against those of the previous rule only (in other + // words, the input is assumed to be ordered). + void AddRule(const StringPiece &target, const StringPiece &ntAlign, + const StringPiece &fullAlign, int count, double treeScore); + +private: + std::string m_source; + std::vector m_distinctRules; + int m_totalCount; +}; + +} // namespace ScoreStsg +} // namespace Syntax +} // namespace MosesTraining diff --git a/mosesdecoder/phrase-extract/score-stsg/RuleTableWriter.cpp b/mosesdecoder/phrase-extract/score-stsg/RuleTableWriter.cpp new file mode 100644 index 0000000000000000000000000000000000000000..d7bbe9d7f298ac7a74de803783b650966200f824 --- /dev/null +++ b/mosesdecoder/phrase-extract/score-stsg/RuleTableWriter.cpp @@ -0,0 +1,82 @@ +#include "RuleTableWriter.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "util/string_piece.hh" +#include "util/tokenize_piece.hh" + +#include "InputFileStream.h" +#include "LexicalTable.h" +#include "OutputFileStream.h" +#include "Options.h" +#include "RuleGroup.h" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace ScoreStsg +{ + +void RuleTableWriter::WriteLine(const TokenizedRuleHalf &source, + const TokenizedRuleHalf &target, + const std::string &bestAlignment, + double lexScore, double treeScore, int count, + int totalCount, int distinctCount) +{ + if (m_options.inverse) { + WriteRuleHalf(target); + m_out << " ||| "; + WriteRuleHalf(source); + } else { + WriteRuleHalf(source); + m_out << " ||| "; + WriteRuleHalf(target); + } + + m_out << " |||" << bestAlignment << "||| "; + + if (!m_options.noLex) { + m_out << MaybeLog(lexScore); + } + + if (m_options.treeScore && !m_options.inverse) { + m_out << " " << MaybeLog(treeScore); + } + + m_out << " ||| " << totalCount << " " << count; + if (m_options.kneserNey) { + m_out << " " << distinctCount; + } + m_out << " |||"; + m_out << std::endl; +} + +void RuleTableWriter::WriteRuleHalf(const TokenizedRuleHalf &half) +{ + if (half.IsTree()) { + m_out << half.string; + return; + } + + for (std::vector::const_iterator p = half.frontierSymbols.begin(); + p != half.frontierSymbols.end(); ++p) { + if (p->isNonTerminal) { + m_out << "[" << p->value << "][" << p->value << "] "; + } else { + m_out << p->value << " "; + } + } + m_out << "[X]"; +} + +} // namespace ScoreStsg +} // namespace Syntax +} // namespace MosesTraining diff --git a/mosesdecoder/phrase-extract/score-stsg/TokenizedRuleHalf.h b/mosesdecoder/phrase-extract/score-stsg/TokenizedRuleHalf.h new file mode 100644 index 0000000000000000000000000000000000000000..7d2b742168c464e154cd4e0d875ca756a4c8e1ee --- /dev/null +++ b/mosesdecoder/phrase-extract/score-stsg/TokenizedRuleHalf.h @@ -0,0 +1,48 @@ +#pragma once + +#include +#include + +#include "syntax-common/tree_fragment_tokenizer.h" + +#include "RuleSymbol.h" + +namespace MosesTraining +{ +namespace Syntax +{ +namespace ScoreStsg +{ + +// Stores one half of a STSG rule, as represented in the extract file. The +// original string is stored as the member 'string', along with its token +// sequence ('tokens') and frontier symbol sequence ('frontierSymbols'). Note +// that 'tokens' and 'frontierSymbols' use StringPiece objects that depend on +// the original string. Therefore changing the value of 'string' invalidates +// both 'tokens' and 'frontierSymbols'. +struct TokenizedRuleHalf { + bool IsFullyLexical() const; + bool IsString() const; + bool IsTree() const; + + // The rule half as it appears in the extract file, except with any trailing + // or leading spaces removed (here a space is defined as a blank or a tab). + std::string string; + + // The token sequence for the string. + std::vector tokens; + + // The frontier symbols of the rule half. For example: + // + // string: "[VP [VBN] [PP [IN] [NP [DT] [JJ positive] [NN light]]]]" + // frontier: ("VBN",t), ("IN",t), ("DT",t), ("positive",f), ("light",f) + // + // string: "[X] [X] Sinne [X]" + // frontier: ("X",t), ("X",t), ("Sinne",f), ("X",t) + // + std::vector frontierSymbols; +}; + +} // namespace ScoreStsg +} // namespace Syntax +} // namespace MosesTraining