diff --git a/.gitattributes b/.gitattributes index a6344aac8c09253b3b630fb776ae94478aa0275b..e8d8fd8de4ba804a53e69255799d5d7df127e9f1 100644 --- a/.gitattributes +++ b/.gitattributes @@ -33,3 +33,17 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text *.zip filter=lfs diff=lfs merge=lfs -text *.zst filter=lfs diff=lfs merge=lfs -text *tfevents* filter=lfs diff=lfs merge=lfs -text +perception_models/apps/pe/docs/assets/dog.mp4 filter=lfs diff=lfs merge=lfs -text +perception_models/apps/pe/docs/assets/dog.png filter=lfs diff=lfs merge=lfs -text +perception_models/apps/pe/docs/assets/office.mp4 filter=lfs diff=lfs merge=lfs -text +perception_models/apps/pe/docs/assets/office.wav filter=lfs diff=lfs merge=lfs -text +perception_models/apps/pe/docs/assets/pikachu.webp filter=lfs diff=lfs merge=lfs -text +perception_models/apps/pe/docs/assets/shark.png filter=lfs diff=lfs merge=lfs -text +perception_models/apps/pe/docs/assets/spatial_correspondence.png filter=lfs diff=lfs merge=lfs -text +perception_models/apps/pe/docs/assets/spatial_features.png filter=lfs diff=lfs merge=lfs -text +perception_models/apps/pe/docs/assets/teaser.png filter=lfs diff=lfs merge=lfs -text +perception_models/apps/pe/docs/assets/train.mp4 filter=lfs diff=lfs merge=lfs -text +perception_models/apps/pe/docs/assets/train.wav filter=lfs diff=lfs merge=lfs -text +perception_models/apps/plm/docs/plm_main_fig.png filter=lfs diff=lfs merge=lfs -text +perception_models/core/tests/Rock-climbing-Canada-1920x1147.jpg filter=lfs diff=lfs merge=lfs -text +perception_models/core/tests/selfie_cathedral_peak.jpg filter=lfs diff=lfs merge=lfs -text diff --git a/perception_models/.gitignore b/perception_models/.gitignore new file mode 100644 index 0000000000000000000000000000000000000000..42a1a625a22e9e315ddffcfa656c3f311576a1c9 --- /dev/null +++ b/perception_models/.gitignore @@ -0,0 +1,10 @@ +*.pyc +.vscode +*.ipynb +slurm-*.out +wandb +data/* +data-gym-cache/* +torchinductor_*/* +tmp*/* +apps/plm/dummy_datasets diff --git a/perception_models/CODE_OF_CONDUCT.md b/perception_models/CODE_OF_CONDUCT.md new file mode 100644 index 0000000000000000000000000000000000000000..08b500a221857ec3f451338e80b4a9ab1173a1af --- /dev/null +++ b/perception_models/CODE_OF_CONDUCT.md @@ -0,0 +1,80 @@ +# Code of Conduct + +## Our Pledge + +In the interest of fostering an open and welcoming environment, we as +contributors and maintainers pledge to make participation in our project and +our community a harassment-free experience for everyone, regardless of age, body +size, disability, ethnicity, sex characteristics, gender identity and expression, +level of experience, education, socio-economic status, nationality, personal +appearance, race, religion, or sexual identity and orientation. + +## Our Standards + +Examples of behavior that contributes to creating a positive environment +include: + +* Using welcoming and inclusive language +* Being respectful of differing viewpoints and experiences +* Gracefully accepting constructive criticism +* Focusing on what is best for the community +* Showing empathy towards other community members + +Examples of unacceptable behavior by participants include: + +* The use of sexualized language or imagery and unwelcome sexual attention or + advances +* Trolling, insulting/derogatory comments, and personal or political attacks +* Public or private harassment +* Publishing others' private information, such as a physical or electronic + address, without explicit permission +* Other conduct which could reasonably be considered inappropriate in a + professional setting + +## Our Responsibilities + +Project maintainers are responsible for clarifying the standards of acceptable +behavior and are expected to take appropriate and fair corrective action in +response to any instances of unacceptable behavior. + +Project maintainers have the right and responsibility to remove, edit, or +reject comments, commits, code, wiki edits, issues, and other contributions +that are not aligned to this Code of Conduct, or to ban temporarily or +permanently any contributor for other behaviors that they deem inappropriate, +threatening, offensive, or harmful. + +## Scope + +This Code of Conduct applies within all project spaces, and it also applies when +an individual is representing the project or its community in public spaces. +Examples of representing a project or community include using an official +project e-mail address, posting via an official social media account, or acting +as an appointed representative at an online or offline event. Representation of +a project may be further defined and clarified by project maintainers. + +This Code of Conduct also applies outside the project spaces when there is a +reasonable belief that an individual's behavior may have a negative impact on +the project or its community. + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported by contacting the project team at . All +complaints will be reviewed and investigated and will result in a response that +is deemed necessary and appropriate to the circumstances. The project team is +obligated to maintain confidentiality with regard to the reporter of an incident. +Further details of specific enforcement policies may be posted separately. + +Project maintainers who do not follow or enforce the Code of Conduct in good +faith may face temporary or permanent repercussions as determined by other +members of the project's leadership. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, +available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html + +[homepage]: https://www.contributor-covenant.org + +For answers to common questions about this code of conduct, see +https://www.contributor-covenant.org/faq diff --git a/perception_models/CONTRIBUTING.md b/perception_models/CONTRIBUTING.md new file mode 100644 index 0000000000000000000000000000000000000000..fd6fbec6d5ad44923bb1bc206b33f5ca9e8b67f3 --- /dev/null +++ b/perception_models/CONTRIBUTING.md @@ -0,0 +1,31 @@ +# Contributing to Perception Models +We want to make contributing to this project as easy and transparent as +possible. + +## Pull Requests +We actively welcome your pull requests. + +1. Fork the repo and create your branch from `main`. +2. If you've added code that should be tested, add tests. +3. If you've changed APIs, update the documentation. +4. Ensure the test suite passes. +5. Make sure your code lints. +6. If you haven't already, complete the Contributor License Agreement ("CLA"). + +## Contributor License Agreement ("CLA") +In order to accept your pull request, we need you to submit a CLA. You only need +to do this once to work on any of Facebook's open source projects. + +Complete your CLA here: + +## Issues +We use GitHub issues to track public bugs. Please ensure your description is +clear and has sufficient instructions to be able to reproduce the issue. + +Facebook has a [bounty program](https://www.facebook.com/whitehat/) for the safe +disclosure of security bugs. In those cases, please go through the process +outlined on that page and do not file a public issue. + +## License +By contributing to mae, you agree that your contributions will be licensed +under the LICENSE file in the root directory of this source tree. diff --git a/perception_models/LEGRAD_PE_USAGE.md b/perception_models/LEGRAD_PE_USAGE.md new file mode 100644 index 0000000000000000000000000000000000000000..41455f2a0905d262e2174b6680ef401896775e1c --- /dev/null +++ b/perception_models/LEGRAD_PE_USAGE.md @@ -0,0 +1,72 @@ +# LeGrad + PE Perception Encoder Notebook Usage + +This repository includes a notebook `legrad_perception_encoder.ipynb` that demonstrates how to run **LeGrad** explanations on the PE CoCa-style vision encoder. + +## 1. Environment and installation + +- **Install this repo** (from the repo root): + +```bash +pip install -e . +``` + +- **Install LeGrad** (if not already installed): + +```bash +pip install legrad +``` + +Make sure you have a working CUDA‑enabled PyTorch environment. + +## 2. Open the notebook + +From the repo root: + +```bash +cd xai/perception_models +jupyter lab legrad_perception_encoder.ipynb +``` + +## 3. What the notebook does + +The notebook shows how to: + +1. Load a PE CoCa‑style vision encoder: + - Uses `pe.CLIP.from_config("PE-Core-B16-224", pretrained=True)` and moves the model to CUDA. +2. Wrap the model with LeGrad: + - `LeWrapper` lives in `core/legrad_pe.py`. + - It hooks PE residual blocks and attention pooling so gradients can be used to build visual explanations. +3. Prepare inputs: + - Build an image transform with `transforms.get_image_transform(model.image_size)`. + - Tokenize text prompts with `transforms.get_text_tokenizer(model.context_length)`. +4. Run LeGrad: + - **Multi‑layer explanation**: + - `heatmap = wrapped_model.compute_legrad_coca(text_emb, image=image_tensor)` + - **Single‑layer explanation**: + - `heatmap = wrapped_model.compute_legrad_coca_one_layer(text_emb, image=image_tensor, layer_idx=-1)` +5. Visualize: + - Convert the `heatmap` to numpy and use `legrad.visualize` (or standard plotting) to overlay it on the image. + +## 4. Minimal code sketch (inside the notebook) + +The core usage pattern is: + +```python +import core.vision_encoder.pe as pe +import core.vision_encoder.transforms as transforms +from core.legrad_pe import LeWrapper + +model = pe.CLIP.from_config("PE-Core-B16-224", pretrained=True).cuda() +preprocess = transforms.get_image_transform(model.image_size) +tokenizer = transforms.get_text_tokenizer(model.context_length) + +wrapped_model = LeWrapper(model, layer_index=-2) +``` + +You can then: + +- Preprocess an input image with `preprocess`, +- Tokenize prompts with `tokenizer`, +- Encode text/image, and +- Call one of the `compute_legrad_*` methods to obtain a heatmap for visualization. + diff --git a/perception_models/LICENSE.PE b/perception_models/LICENSE.PE new file mode 100644 index 0000000000000000000000000000000000000000..261eeb9e9f8b2b4b0d119366dda99c6fd7d35c64 --- /dev/null +++ b/perception_models/LICENSE.PE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/perception_models/LICENSE.PLM b/perception_models/LICENSE.PLM new file mode 100644 index 0000000000000000000000000000000000000000..96372f90a1b9be33e955d1b0482534f0651aee84 --- /dev/null +++ b/perception_models/LICENSE.PLM @@ -0,0 +1,124 @@ +FAIR Noncommercial Research License +Last Updated: 17 April 2025 + +“Acceptable Use Policy” means the FAIR Acceptable Use Policy, applicable to Research Materials, that is incorporated into this Agreement. + +“Agreement” means the terms and conditions for use, reproduction, distribution and modification of the Research Materials set forth herein. + + +“Documentation” means the specifications, manuals and documentation accompanying +Research Materials distributed by Meta. + + +“Licensee” or “you” means you, or your employer or any other person or entity (if you are entering into this Agreement on such person or entity’s behalf), of the age required under applicable laws, rules or regulations to provide legal consent and that has legal authority to bind your employer or such other person or entity if you are entering in this Agreement on their behalf. + + +“Meta” or “we” means Meta Platforms Ireland Limited (if you are located in or, if you are an entity, your principal place of business is in the EEA or Switzerland) and Meta Platforms, Inc. (if you are located outside of the EEA or Switzerland). + +“Noncommercial Research Uses” means noncommercial research use cases related to research, development, education, processing, or analysis and in each case, is not primarily intended for commercial advantage or monetary compensation to you or others. + +“Research Materials” means, collectively, Documentation and the models, software and algorithms, including machine-learning model code, trained model weights, inference-enabling code, training-enabling code, fine-tuning enabling code, demonstration materials and other elements of the foregoing distributed by Meta and made available under this Agreement. + +By clicking “I Accept” below or by using or distributing any portion or element of the Research Materials, you agree to be bound by this Agreement. + + +1. License Rights and Redistribution. + + +a. Grant of Rights. You are granted a non-exclusive, worldwide, non-transferable and royalty-free limited license under Meta’s intellectual property or other rights owned by Meta embodied in the Research Materials to use, reproduce, distribute, copy, create derivative works of, and make modifications to the Research Materials. + +b. Redistribution and Use. + i. You will not use the Research Materials or any outputs or results of the Research Materials in connection with any commercial uses or for any uses other than Noncommercial Research Uses; + + +ii. Distribution of Research Materials, and any derivative works thereof, are subject to the terms of this Agreement. If you distribute or make the Research Materials, or any derivative works thereof, available to a third party, you may only do so under the terms of this Agreement. You shall also provide a copy of this Agreement to such third party. + + +iii. If you submit for publication the results of research you perform on, using, or otherwise in connection with Research Materials, you must acknowledge the use of Research Materials in your publication. + + +iv. Your use of the Research Materials must comply with applicable laws and regulations (including Trade Control Laws) and adhere to the FAIR Acceptable Use Policy, which is hereby incorporated by reference into this Agreement. +2. User Support. Your Noncommercial Research Use of the Research Materials is done at your own discretion; Meta does not process any information nor provide any service in relation to such use. Meta is under no obligation to provide any support services for the Research Materials. Any support provided is “as is”, “with all faults”, and without warranty of any kind. + + +3. Disclaimer of Warranty. UNLESS REQUIRED BY APPLICABLE LAW, THE RESEARCH MATERIALS AND ANY OUTPUT AND RESULTS THEREFROM ARE PROVIDED ON AN “AS IS” BASIS, WITHOUT WARRANTIES OF ANY KIND, AND META DISCLAIMS ALL WARRANTIES OF ANY KIND, BOTH EXPRESS AND IMPLIED, INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. YOU ARE SOLELY RESPONSIBLE FOR DETERMINING THE APPROPRIATENESS OF USING OR REDISTRIBUTING THE RESEARCH MATERIALS AND ASSUME ANY RISKS ASSOCIATED WITH YOUR USE OF THE RESEARCH MATERIALS AND ANY OUTPUT AND RESULTS. + +4. Limitation of Liability. IN NO EVENT WILL META OR ITS AFFILIATES BE LIABLE UNDER ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, TORT, NEGLIGENCE, PRODUCTS LIABILITY, OR OTHERWISE, ARISING OUT OF THIS AGREEMENT, FOR ANY LOST PROFITS OR ANY DIRECT OR INDIRECT, SPECIAL, CONSEQUENTIAL, INCIDENTAL, EXEMPLARY OR PUNITIVE DAMAGES, EVEN IF META OR ITS AFFILIATES HAVE BEEN ADVISED OF THE POSSIBILITY OF ANY OF THE FOREGOING. + +5. Intellectual Property. + + +a. Subject to Meta’s ownership of Research Materials and derivatives made by or for Meta, with respect to any derivative works and modifications of the Research Materials that are made by you, as between you and Meta, you are and will be the owner of such derivative works and modifications. + +b. If you institute litigation or other proceedings against Meta or any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Research Materials, outputs or results, or any portion of any of the foregoing, constitutes infringement of intellectual property or other rights owned or licensable by you, then any licenses granted to you under this Agreement shall terminate as of the date such litigation or claim is filed or instituted. You will indemnify and hold harmless Meta from and against any claim by any third party arising out of or related to your use or distribution of the Research Materials. + +6. Term and Termination. The term of this Agreement will commence upon your acceptance of this Agreement or access to the Research Materials and will continue in full force and effect until terminated in accordance with the terms and conditions herein. Meta may terminate this Agreement if you are in breach of any term or condition of this Agreement. Upon termination of this Agreement, you shall delete and cease use of the Research Materials. Sections 5, 6 and 9 shall survive the termination of this Agreement. + +7. Governing Law and Jurisdiction. This Agreement will be governed and construed under the laws of the State of California without regard to choice of law principles, and the UN Convention on Contracts for the International Sale of Goods does not apply to this Agreement. The courts of California shall have exclusive jurisdiction of any dispute arising out of this Agreement. + + +8. Modifications and Amendments. Meta may modify this Agreement from time to time by posting a revised version at [INSERT URL]; provided that they are similar in spirit to the current version of the Agreement, but may differ in detail to address new problems or concerns. All such changes will be effective immediately. Your continued use of the Research Materials after any modification to this Agreement constitutes your agreement to such modification. Except as provided in this Agreement, no modification or addition to any provision of this Agreement will be binding unless it is in writing and signed by an authorized representative of both you and Meta. + + +FAIR Acceptable Use Policy + +The Fundamental AI Research (FAIR) team at Meta seeks to further understanding of new and existing research domains with the mission of advancing the state-of-the-art in artificial intelligence through open research for the benefit of all. + +As part of this mission, Meta makes certain research materials available for noncommercial research use. Meta is committed to promoting the safe and responsible use of such research materials. + +Prohibited Uses + +You agree you will not use, or allow others to use, Research Materials to: + + Violate the law or others’ rights, including to: +Engage in, promote, generate, contribute to, encourage, plan, incite, or further illegal or unlawful activity or content, such as: +Violence or terrorism +Exploitation or harm to children, including the solicitation, creation, acquisition, or dissemination of child exploitative content or failure to report Child Sexual Abuse Material +Human trafficking, exploitation, and sexual violence +The illegal distribution of information or materials to minors, including obscene materials, or failure to employ legally required age-gating in connection with such information or materials. +Sexual solicitation +Any other criminal activity + +Engage in, promote, incite, or facilitate the harassment, abuse, threatening, or bullying of individuals or groups of individuals + +Engage in, promote, incite, or facilitate discrimination or other unlawful or harmful conduct in the provision of employment, employment benefits, credit, housing, other economic benefits, or other essential goods and services + +Engage in the unauthorized or unlicensed practice of any profession including, but not limited to, financial, legal, medical/health, or related professional practices + +Collect, process, disclose, generate, or infer health, demographic, or other sensitive personal or private information about individuals without rights and consents required by applicable laws + +Engage in or facilitate any action or generate any content that infringes, misappropriates, or otherwise violates any third-party rights, including the outputs or results of any technology using FAIR research materials + +Create, generate, or facilitate the creation of malicious code, malware, computer viruses or do anything else that could disable, overburden, interfere with or impair the proper working, integrity, operation or appearance of a website or computer system + +2. Engage in, promote, incite, facilitate, or assist in the planning or development of activities that present a risk of death or bodily harm to individuals, including use of research artifacts related to the following: + +Military, warfare, nuclear industries or applications, espionage, use for materials or activities that are subject to the International Traffic Arms Regulations (ITAR) maintained by the United States Department of State + +Guns and illegal weapons (including weapon development) + +Illegal drugs and regulated/controlled substances + +Operation of critical infrastructure, transportation technologies, or heavy machinery + +Self-harm or harm to others, including suicide, cutting, and eating disorders + +Any content intended to incite or promote violence, abuse, or any infliction of bodily harm to an individual + +3. Intentionally deceive or mislead others, including use of FAIR Research Materials related to the following: + + Generating, promoting, or furthering fraud or the creation or promotion of disinformation + + Generating, promoting, or furthering defamatory content, including the creation of defamatory statements, images, or other content + +Generating, promoting, or further distributing spam + + Impersonating another individual without consent, authorization, or legal right + +Representing that outputs of FAIR research materials or outputs from technology using FAIR research materials are human-generated + +Generating or facilitating false online engagement, including fake reviews and other means of fake online engagement + +4. Fail to appropriately disclose to end users any known dangers of your Research Materials. + +Please report any violation of this Policy or other problems that could lead to a violation of this Policy by submitting a report here [https://docs.google.com/forms/d/e/1FAIpQLSeb11cryAopJ7LNrC4nxEUXrHY26hfkXQMf_uH-oFgA3WlYZQ/viewform]. diff --git a/perception_models/README.md b/perception_models/README.md new file mode 100644 index 0000000000000000000000000000000000000000..793d92f130af7305fcce7a28044881a6bc76ad08 --- /dev/null +++ b/perception_models/README.md @@ -0,0 +1,408 @@ +# Perception Models: Powerful Models for Image, Video, and Audio Perception +[![Code License](https://img.shields.io/badge/Code_License-Apache_2.0-olive)](https://opensource.org/licenses/Apache-2.0) + +This repo is the home to the state-of-the-art for image and video _perception_: [**Perception Encoder (PE)**](https://arxiv.org/abs/2504.13181) for image, video, [audio](https://ai.meta.com/research/publications/pushing-the-frontier-of-audiovisual-perception-with-large-scale-multimodal-correspondence-learning/) encoding, and [**Perception Language Model (PLM)**](https://arxiv.org/abs/2504.13180) for decoding. + +> [!TIP] +> Click to Navigate! +> +> [Perception Encoder and Perception Encoder Audio-Visual](#perception-encoder-pe) +> +> [Perception Language Model](#perception-language-model-plm) +> +> [Dataset Releases](#dataset-releases) + +## Updates +* **[Dec-16-25]:** We have released the Perception Encoder Audio-Visual (PE-AV) and Perception Encoder Audio-Frame (PE-A-Frame) models: [[`Blog`](https://ai.meta.com/blog/sam-audio/)][[`paper`](https://ai.meta.com/research/publications/pushing-the-frontier-of-audiovisual-perception-with-large-scale-multimodal-correspondence-learning/)] :fire::fire: +* **[Jul-14-25]:** PerceptionLM is now available in [Hugging Face transformers](https://huggingface.co/docs/transformers/main/en/model_doc/perception_lm). :fire::fire: +* **[Jul-11-25]:** We have release 8 new checkpoints for [Perception Encoder](apps/pe/README.md): 2x small core models (T and S), 2x tiling-tuned lang models (G and L), and 4x smaller spatial models (L, B, S, T). Give them a try! :fire::fire::fire: +* **[May-28-25]:** Perception Encoder has been integrated into [timm](https://github.com/huggingface/pytorch-image-models)! :fire::fire: +* **[Apr-18-25]:** Perception Language Model (PLM) and PLM-VideoBench are added to lmms-eval. This makes it easy to reproduce PLM results and allows you to evaluate on the PLM-VideoBench. [[`lmms-eval`](https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/638)] :fire::fire: +* **[Apr-17-25]:** Perception Encoder (PE) and Perception Language Model (PLM) are released. [[`Blog`](https://ai.meta.com/blog/meta-fair-updates-perception-localization-reasoning)] :fire::fire: + + +## Perception Encoder (PE) +[![Data](https://img.shields.io/badge/Download-PE%20Data-ffcc00.svg)](https://huggingface.co/datasets/facebook/PE-Video) +[![Hugging Face Collection](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Collection-blue)](https://huggingface.co/collections/facebook/perception-encoder-67f977c9a65ca5895a7f6ba1) +[![Paper](https://img.shields.io/badge/Technical%20Report-Perception%20Encoder-b31b1b.svg)](https://ai.meta.com/research/publications/perception-encoder-the-best-visual-embeddings-are-not-at-the-output-of-the-network) +[![Paper](https://img.shields.io/badge/Technical%20Report-Perception%20Encoder%20AV-b31b1b.svg)](https://ai.meta.com/research/publications/pushing-the-frontier-of-audiovisual-perception-with-large-scale-multimodal-correspondence-learning/) +[![Paper](https://img.shields.io/badge/arXiv-2504.13181-brightgreen.svg?style=flat-square)](https://arxiv.org/abs/2504.13181) +[![Colab Demo](https://img.shields.io/static/v1?label=Demo&message=Google%20Colab&logo=google&color=orange)](https://colab.research.google.com/github/facebookresearch/perception_models/blob/main/apps/pe/docs/pe_demo.ipynb) +[![Model License](https://img.shields.io/badge/Model_License-Apache_2.0-olive)](https://opensource.org/licenses/Apache-2.0) + +[Perception Encoder (PE)](https://arxiv.org/abs/2504.13181) is a family of the state-of-the-art vision and audio encoders for encoding images, video, and audio: PE core outperforms SigLIP2 on image and InternVideo2 on video benchmarks; PE lang can be used to outperform QwenVL2.5 and InternVL3 on vision language modeling; and PE spatial outperforms DINOv2 on dense prediction tasks. And all of this follows the same, easily scalable contrastive pretraining. Please see [README](apps/pe/README.md) for more details. + + + +### Models +PE has 4 types of checkpoints, each excelling in a different area of computer vision and audio understanding: + - [PE core](#vision-language-benchmarks): a CLIP model excels in vision-language tasks such as zero-shot image and video classification and video retrieval. + - [PE lang](#multimodal-llm-benchmarks): a LLM-aligned PE that powers [PLM](https://arxiv.org/abs/2504.13180) to compete at the forefront of multimodal LLM benchmarks. + - [PE spatial](#vision-centric-benchmarks): a spatially tuned PE that outperforms best spatial models for vision-centric tasks such as detection, depth estimation, and tracking. + - [PE audio-visual](#audio-visual-benchmarks): a CLIP Model that embeds audio, video, audio-video, and text into a joint embedding space. + +#### Vision-Language Benchmarks +| | Model | Checkpoint | IN-1k | IN-v2 | IN-A | ObjectNet | COCO-T2I | Kinetics-400 | VTT-T2V +|:--:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| +| | **T/16** 384px | [PE-Core-T16-384](https://huggingface.co/facebook/PE-Core-T16-384) | 62.1 | 54.7 | 21.1 | 43.9 | 33.0 | 41.5 | 28.8 | +| | **S/16** 384px | [PE-Core-S16-384](https://huggingface.co/facebook/PE-Core-S16-384) | 72.7 | 65.0 | 49.5 | 60.0 | 42.6 | 55.0 | 39.3 | +| | **B/16** 224px | [PE-Core-B16-224](https://huggingface.co/facebook/PE-Core-B16-224) | 78.4 | 71.7 | 62.4 | 71.9 | 50.9 | 65.6 | 47.6 | +| | **L/14** 336px | [PE-Core-L14-336](https://huggingface.co/facebook/PE-Core-L14-336) | 83.5 | 77.9 | 89.0 | 84.7 | 57.1 | 73.4 | 50.3 | +| | **G/14** 448px | [PE-Core-G14-448](https://huggingface.co/facebook/PE-Core-G14-448) | 85.4 | 80.2 | 92.6 | 88.2 | 58.1 | 76.9 | 51.2 | + +#### Multimodal LLM Benchmarks + +🔬 Controlled Setting: +| | Encoder | Checkpoint | Doc VQA (val) | InfoQA (val) | TextVQA | MVBench | PerceptionTest (val) | EgoSchema (val) | +|:--:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| +| | **L/14** 448px | [PE-Lang-L14-448](https://huggingface.co/facebook/PE-Lang-L14-448) | 81.9 | 46.4 | 73.0 | 52.3 | 54.7 | 59.8 | +| | **G/14** 448px | [PE-Lang-G14-448](https://huggingface.co/facebook/PE-Lang-G14-448) | 84.4 | 48.3 | 75.2 | 52.4 | 56.0 | 62.0 | + + +🔥 SotA Setting: +| | Model | Encoder | Doc VQA (test) | InfoQA (test) | TextVQA | MVBench | PerceptionTest (test) | EgoSchema (test) | +|:--:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| +| | PLM-3B | [PE-Lang-L14-448-Tiling](https://huggingface.co/facebook/PE-Lang-L14-448-Tiling)* | 93.8 | 74.6 | 84.3 | 74.7 | 79.3 | 66.9 | +| | PLM-8B | [PE-Lang-G14-448-Tiling](https://huggingface.co/facebook/PE-Lang-G14-448-Tiling)* | 94.6 | 80.9 | 86.5 | 77.1 | 82.7 | 68.8 | + +\* These checkpoints were aligned with tiling. Use them if you use higher than 448 resolution with tiling in the LLM decoder. + +#### Vision-centric Benchmarks +🦾 Main model: +| | Encoder | Checkpoint | ADE20k
[Segmentation](https://github.com/open-mmlab/mmsegmentation)
Linear Probe mIoU | DAVIS
[Tracking](https://github.com/facebookresearch/dino/blob/main/eval_video_segmentation.py)
Zero-Shot J&F | LVIS
[Mask R-CNN](../detection/detectron2_pe/) 1024px
Box / Mask mAP | COCO
[DETA](../detection/DETA_pe/) 1824px
Box mAP | +|:--:|:---:|:---:|:---:|:---:|:---:|:---:| +| | **G/14** 448px | [PE-Spatial-G14-448](https://huggingface.co/facebook/PE-Spatial-G14-448) | 49.3 | 61.5 | 54.2 / 49.3 | 66.0 | + + +
+ + + Visualization of PCA of non-maked visual tokens, mapped to RGB values. +
+ +⚗️ Distilled Models: +| | Encoder
(Distilled from G) | Checkpoint | ADE20k
[Segmentation](https://github.com/open-mmlab/mmsegmentation)
Linear Probe mIoU | DAVIS
[Tracking](https://github.com/facebookresearch/dino/blob/main/eval_video_segmentation.py)
Zero-Shot J&F | +|:--:|:---:|:---:|:---:|:---:| +| | **T/16** 512px | [PE-Spatial-T16-512](https://huggingface.co/facebook/PE-Spatial-T16-512) | 27.6 | 55.0 | +| | **S/16** 512px | [PE-Spatial-S16-512](https://huggingface.co/facebook/PE-Spatial-S16-512) | 37.5 | 57.5 | +| | **B/16** 512px | [PE-Spatial-B16-512](https://huggingface.co/facebook/PE-Spatial-B16-512) | 44.4 | 58.9 | +| | **L/14** 448px | [PE-Spatial-L14-448](https://huggingface.co/facebook/PE-Spatial-L14-448) | 48.1 | 60.6 | + +See paper for comparison to other models. + +#### Audio-Visual Benchmarks + +| | Model | Checkpoint | Avg Retrieval | AudioCaps T→A | AudioCaps T→V | AudioCaps V→A | Clotho T→A | Valor T→A | Valor T→V | VCTK A→T | VGGSound V→A | Internal V→A | +|:--:|:-----:|--------------|---------------|---------------|---------------|---------------|------------|-----------|-----------|----------|---------------|---------------| +| 🆕 | **AV S** 16 frames | [`pe-av-small-16-frame`](https://huggingface.co/facebook/pe-av-small-16-frame) | 45.2 | 41.2 | 18.6 | 75.4 | 24.0 | 29.8 | 70.1 | 96.1 | 34.1 | 17.9 | +| 🆕 | **AV B** 16 frames | [`pe-av-base-16-frame`](https://huggingface.co/facebook/pe-av-base-16-frame) | 47.0 | 43.1 | 19.8 | 80.6 | 23.4 | 31.9 | 70.0 | 94.8 | 39.0 | 20.4 | +| 🆕 | **AV L** 16 frames | [`pe-av-large-16-frame`](https://huggingface.co/facebook/pe-av-large-16-frame) | 48.2 | 44.7 | 19.5 | 86.1 | 22.8 | 35.0 | 70.9 | 85.6 | 45.2 | 23.9 | +| 🆕 | **AV S** all frames | [`pe-av-small`](https://huggingface.co/facebook/pe-av-small) | 48.1 | 41.8 | 18.8 | 77.4 | 23.9 | 29.3 | 70.9 | 94.9 | 35.4 | 40.5 | +| 🆕 | **AV B** all frames | [`pe-av-base`](https://huggingface.co/facebook/pe-av-base) | 50.2 | 42.7 | 19.6 | 83.7 | 23.8 | 30.8 | 71.2 | 94.9 | 40.7 | 44.6 | +| 🆕 | **AV L** all frames | [`pe-av-large`](https://huggingface.co/facebook/pe-av-large) | 51.6 | 45.8 | 20.8 | 88.3 | 23.0 | 35.1 | 70.9 | 85.6 | 48.3 | 46.5 | + +#### Audio Event Localization Benchmarks + +| | Model | Checkpoint | Internal Bench (AUROC) | ASFX-SED (AUROC) | AudioSet-Strong (AUROC) | DESED (AUROC) | UrbanSED (AUROC) | +|:--:|:-----:|------------------|---------------------|------------------|-----------------------|-------------|-------------| +| 🆕 | **A-Frame S** | [`pe-a-frame-small`](https://huggingface.co/facebook/pe-a-frame-small)| 0.91 | 0.83 | 0.96 | 0.96 | 0.88 | +| 🆕 | **A-Frame B** | [`pe-a-frame-base`](https://huggingface.co/facebook/pe-a-frame-base)| 0.92 | 0.83 | 0.96 | 0.98 | 0.89 | +| 🆕 | **A-Frame L** | [`pe-a-frame-large`](https://huggingface.co/facebook/pe-a-frame-large)| 0.91 | 0.83 | 0.96 | 0.97 | 0.89 | + +### Getting Started with PE +You can get started with the following example for image and text feature extraction or use our [Colab Demo](https://colab.research.google.com/github/facebookresearch/perception_models/blob/main/apps/pe/docs/pe_demo.ipynb) + +```python +import torch +from PIL import Image +import core.vision_encoder.pe as pe +import core.vision_encoder.transforms as transforms + +print("CLIP configs:", pe.CLIP.available_configs()) +# CLIP configs: ['PE-Core-G14-448', 'PE-Core-L14-336', 'PE-Core-B16-224', 'PE-Core-S16-384', 'PE-Core-T16-384'] + +model = pe.CLIP.from_config("PE-Core-L14-336", pretrained=True) # Downloads from HF +model = model.cuda() + +preprocess = transforms.get_image_transform(model.image_size) +tokenizer = transforms.get_text_tokenizer(model.context_length) + +image = preprocess(Image.open("docs/assets/cat.png")).unsqueeze(0).cuda() +text = tokenizer(["a diagram", "a dog", "a cat"]).cuda() + +with torch.no_grad(), torch.autocast("cuda"): + image_features, text_features, logit_scale = model(image, text) + text_probs = (logit_scale * image_features @ text_features.T).softmax(dim=-1) + +print("Label probs:", text_probs) # prints: [[0.0, 0.0, 1.0]] +``` + +> [!TIP] +> See [`apps/pe/README.md`](apps/pe/README.md) for details and how to get started! + +### Getting Started with PE-AV + +```python +import os +from core.audio_visual_encoder import PEAudioVisual, PEAudioVisualTransform +import torch + +device = torch.device("cuda" if torch.cuda.is_available() else "cpu") +model = PEAudioVisual.from_config("pe-av-large", pretrained=True).to(device) +transform = PEAudioVisualTransform.from_config("pe-av-large") + +video_files = ["assets/train.mp4", "assets/office.mp4"] +descriptions = [ + "A person talking with sirens and a train in the background", + "Two people talking in an office, with sounds of workers typing on a keyboard" +] + +def embed(videos=None, audio=None, text=None): + inputs = transform(videos=videos, audio=audio, text=text) + inputs = inputs.to(device) + with torch.inference_mode(), torch.autocast(device.type, dtype=torch.bfloat16): + return model(**inputs) + +vt_outputs = embed(videos=video_files, text=descriptions) +avt_outputs = embed(videos=video_files, audio=video_files, text=descriptions) +at_outputs = embed(audio=video_files, text=descriptions) + +# Compute dot product between visual and text +vt_dot_products = torch.einsum("ij,ij->i", vt_outputs.visual_embeds, vt_outputs.visual_text_embeds) +# Compute dot product between audio_visual and text +avt_dot_products = torch.einsum("ij,ij->i", avt_outputs.audio_visual_embeds, avt_outputs.audio_visual_text_embeds) +# Compute dot product between audio and text +at_dot_products = torch.einsum("ij,ij->i", at_outputs.audio_embeds, at_outputs.audio_text_embeds) +# Compute dot product between audio and video +av_dot_products = torch.einsum("ij,ij->i", avt_outputs.audio_embeds, avt_outputs.video_embeds) +``` + +### Getting Started with PE-A-Frame + +```python +from core.audio_visual_encoder import ( + PEAudioFrame, + PEAudioFrameTransform, +) +import torch + +device = torch.device("cuda" if torch.cuda.is_available() else "cpu") +model = PEAudioFrame.from_config("pe-a-frame-large", pretrained=True).to(device) +transform = PEAudioFrameTransform.from_config("pe-a-frame-large") + +descriptions = ["a person talking"] +inputs = transform( + audio=["assets/office.mp4"], + text=descriptions, +).to(device) + +with torch.inference_mode(): + outputs = model(**inputs) + +# Print the spans for each description (start and end timestamps for when they occur in the audio) +for description, spans in zip(descriptions, outputs.spans): + span_str = ", ".join([f"({start:.2f}, {end:.2f})" for start, end in spans]) + print(f'"{description}": [{span_str}]') + +``` + +> [!TIP] +> See [`apps/pe/README.md`](apps/pe/README.md) for additional details! + +## Perception Language Model (PLM) +[![Data](https://img.shields.io/badge/Download-PLM%20Data-ffcc00.svg)](https://huggingface.co/datasets/facebook/PLM-Video-Human) +[![Hugging Face Collection](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Collection-blue)](https://huggingface.co/collections/facebook/perception-lm-67f9783f171948c383ee7498) +[![Paper](https://img.shields.io/badge/Technical%20Report-PerceptionLM-b31b1b.svg)](https://ai.meta.com/research/publications/perceptionlm-open-access-data-and-models-for-detailed-visual-understanding) +[![Paper](https://img.shields.io/badge/arXiv-2504.13180-brightgreen.svg?style=flat-square)](https://arxiv.org/abs/2504.13180) +[![Colab](https://img.shields.io/badge/Google%20Colab-Tutorials-red)](apps/plm/notebook_demos) +[![ModelLicense](https://img.shields.io/badge/Model_License-FAIR_Research_License-lightgrey)](LICENSE.PLM) + +PerceptionLM (PLM) is a family of open and fully reproducible models to facilitate research in vision-language modeling (VLM). In conjunction with PE, it is powerful enough to compete with the latest state-of-the-art VLMs such as InternVL3 and QwenVL2.5, while using _fully open data_. We also release the largest spatiotemporally annotated video dense captioning and fine-grained human activity recognition datasets to ever exist. + +![Description of the image](apps/plm/docs/plm_main_fig.png) + +### Models +PLM releases models in three different sizes (1B, 3B and 8B). +* [Perception-LM-1B](https://huggingface.co/facebook/Perception-LM-1B): A PLM model trained using Llama-3.2-1B-Instruct base LLM. +* [Perception-LM-3B](https://huggingface.co/facebook/Perception-LM-3B): A PLM model trained using Llama-3.2-3B-Instruct base LLM. +* [Perception-LM-8B](https://huggingface.co/facebook/Perception-LM-8B): A PLM model trained using Llama-3.1-8B-Instruct base LLM. + +#### PLM Image Benchmark Results + +| Model | DocVQA | ChartQA | TextVQA | InfoQA | AI2D | OCRBench | COCO | Nocap | Flickr | MMMU | VQAv2 | OKVQA | VizWiz | MME | SEED | BLINK | CVBench | RealWorldQA | VSR | POPE | +|:---------:|:--------:|:---------:|:---------:|:--------:|:------:|:----------:|:------------:|:-------------:|:--------------:|:------:|:-------:|:--------:|:--------:|:-----:|:------:|:-------:|:----------:|:-------------:|:-----:|:------:| +| PLM1B | 90.7 | 78.6 | 82.1 | 63.0 | 84.9 | 807 | 138.6 | 124.2 | 100.5 | 34.8 | 81.7 | 61.0 | 59.7 | 1603| 76.3 | 46.8 | 73.8 | 67.1 | 68.8| 88.4 | +| PLM3B | 93.8 | 84.3 | 84.3 | 74.6 | 90.9 | 830 | 144.9 | 126.5 | 98.0 | 41.2 | 84.3 | 66.8 | 64.0 | 1879| 78.5 | 55.4 | 81.4 | 72.4 | 80.4| 88.7 | +| PLM8B | 94.6 | 85.5 | 86.5 | 80.9 | 92.7 | 870 | 146.7 | 129.9 | 105.6 | 46.1 | 85.6 | 69.6 | 67.0 | 1989| 79.3 | 56.0 | 81.3 | 75.0 | 82.8| 89.9 | + +#### PLM Video Benchmark Results + +| Model | VATEX | DREAM 1K | How2QA | MVBench | NExTQA | PerceptionTest (test) | STAR | TVQA | VideoMME | TVBench | ActivityNetQA | EgoSchema (test) | TemporalBench | TOMATO | MotionBench (dev) | TempCompass (MCQ) | CGBench (clue) | Charades STA | VideoHallucer | Halluc. EventHallusion | +|:-------------:|:---------------------------:|:-----------------------:|:---------------------:|:-------------:|:-------------:|:--------------------------:|:----------:|:----------:|:----------------:|:-------------:|:--------------------:|:----------------------:|:---------------------:|:------------:|:------------------------:|:-----------------------:|:---------------------:|:-------------------:|:-------------------------------:|:--------------------------------:| +| PLM1B | 92.5 | 34.3 | 86.4 | 70.1 | 80.3 | 72.7 | 83.7 | 50.3 | 49.2 | 50.4 | 62.5 | 60.4 | 18.2 | 25.5 | 52.2 | 64.6 | 43.6 | 55.2 | 49.2 | 79.5 | +| PLM3B | 96.1 | 37.4 | 89.4 | 74.7 | 83.4 | 79.3 | 84.8 | 55.3 | 54.9 | 58.9 | 66.2 | 66.9 | 23.4 | 30.9 | 60.4 | 69.3 | 47.2 | 57.7 | 55.5 | 76.5 | +| PLM8B | 99.7 | 35.9 | 90.7 | 77.1 | 84.1 | 82.7 | 84.9 | 59.3 | 58.3 | 63.5 | 67.3 | 68.8 | 28.3 | 33.2 | 61.4 | 72.7 | 46.4 | 58.6 | 57.7 | 77.3 | + +### PLM Resources + +| Resource | Description | Documentation | +| --- | --- |--------------------------------------------------------| +| **Evaluation** | Evaluation of PLM using lmms-eval | [`docs/evaluation.md`](apps/plm/docs/evaluation.md) | +| **Training / Finetuning** | Training and finetuning instructions for PLM | [`docs/training.md`](apps/plm/docs/training.md) | +| **PLM-VideoBench** | Evaluation on PLM-VideoBench using lmms-eval | [`docs/plm_videobench.md`](apps/plm/docs/plm_videobench.md) | +| **End-to-End Finetuning Example** | End-to-end finetuning example on radiology images | [`docs/finetune_example.md`](apps/plm/docs/finetune_example.md) | +| **Generating Response** | Generate responses using a trained model with `generate.py` | [`generate.py`](apps/plm/generate.py) | + + +> [!TIP] +> See [`apps/plm/README.md`](apps/plm/README.md) for details and how to get started! + +## Dataset Releases + + +### 🎥 [PE-Video-Dataset (PVD)](https://huggingface.co/datasets/facebook/PE-Video) + + +PVD comprises 1M high quality and diverse videos. Among them, 120K videos are accompanied by automated and human-verified annotations. and all videos are accompanied with video description and keywords. The videos are motion-centered, covering both first-person and third-person views with a wide coverage of scenes. + +🔹 [**PVD**](https://huggingface.co/datasets/facebook/PE-Video) - 1M High-Quality Human Annotated Video Dataset + + + + + + + + + +
PVD
+ output_2
+ A person's hands pruning a plant with green leaves. + 		+ output
+ A detailed diorama of a rural landscape featuring a horse-drawn carriage moving along a dirt path +
+ +--- + + +### 🎥 [PLM-Video-Human](https://huggingface.co/datasets/facebook/PLM-Video-Human) + +PLM-Video-Human is a collection of human-annotated resources for training Vision Language Models, focused on detailed video understanding. Training tasks include: + +🔹 [**FGQA**](https://huggingface.co/datasets/facebook/PLM-Video-Human#fine-grained-question-answering-fgqa) — Fine-Grained Question Answering +🔹 [**RTLoc**](https://huggingface.co/datasets/facebook/PLM-Video-Human#region-temporal-localization-rtloc) — Region-Temporal Localization +🔹 [**RCap**](https://huggingface.co/datasets/facebook/PLM-Video-Human#region-video-captioning-rcap) — Region Video Captioning +🔹 [**RDCap**](https://huggingface.co/datasets/facebook/PLM-Video-Human#region-dense-temporal-captioning-rdcap) — Region Dense Temporal Captioning + + + + + + + + + + + + + + + + +
FGQA
+ fgqa +
QuestionAnswer
In what direction do you move the tool while removing the shell?Both clockwise and anticlockwise.
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
STC
+ stc +
Time (s)      Description
[0, 4]The masked subject is a young boy wearing a red jacket and gray pants. He is grasping a monkey bar–like activity in a playground.
[5, 14]He lets go of his hands and runs to the right side of the frame.
[15, 30]The subject is out of frame.
[31, 45]The subject runs back into the frame toward the higher monkey bar in the playground.
[46, 74]He jumps underneath the metal bar and looks up at it. A man wearing a white polo runs toward the subject.
[75, 116]The man in the white polo lifts the subject upward so he can grasp the higher metal bar. The subject holds onto the bar and hangs from it.
+ +--- + +### 🤖 Auto-Generated Datasets + +Sythetic image/video captions and QAs used in PLM, please refer to the paper, Section 3 (PLM), for more details. The sythetic annotations covers: SA1B, Openimages, Obejct365, ArxivQA, UCSF, PDFAcc, YT-1B, Ego4d with captions, YT-1B with MCQAs and Ego4d with QAs. + +🖼️ [**PLM-Image-Auto**](https://huggingface.co/datasets/facebook/PLM-Image-Auto) — Automatically generated image datasets + +📹 [**PLM-Video-Auto**](https://huggingface.co/datasets/facebook/PLM-Video-Auto) — Automatically generated video datasets + + +--- + +## Installation :wrench: +```shell +git clone https://github.com/facebookresearch/perception_models.git +cd perception_models + +conda create --name perception_models python=3.12 +conda activate perception_models + +# Install PyTorch +pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 xformers --index-url https://download.pytorch.org/whl/cu124 + +# We use torchcodec for decoding videos into PyTorch tensors +conda install ffmpeg -c conda-forge +pip install torchcodec==0.1 --index-url=https://download.pytorch.org/whl/cu124 + +pip install -e . +``` +This will install an editable version of repo, allowing you to make changes to the code without needing to reinstall the package every time. + + +## 🙏 Acknowledgement +We are thankful to [Meta Lingua](https://github.com/facebookresearch/lingua) for releasing their code as open-source contributions. The code structure and code implementation of the LLM is directly forked from [Meta Lingua](https://github.com/facebookresearch/lingua). We are also thankful to [Open_CLIP](https://github.com/mlfoundations/open_clip) for open-source contributions in CLIP training, and [CLIP_benchmark](https://github.com/LAION-AI/CLIP_benchmark) for CLIP model evaluation. + + +## 📜 Citation +```BibTeX +@article{bolya2025PerceptionEncoder, + title={Perception Encoder: The best visual embeddings are not at the output of the network}, + author={Daniel Bolya and Po-Yao Huang and Peize Sun and Jang Hyun Cho and Andrea Madotto and Chen Wei and Tengyu Ma and Jiale Zhi and Jathushan Rajasegaran and Hanoona Rasheed and Junke Wang and Marco Monteiro and Hu Xu and Shiyu Dong and Nikhila Ravi and Daniel Li and Piotr Doll{\'a}r and Christoph Feichtenhofer}, + journal={arXiv:2504.13181}, + year={2025} +} + +@article{cho2025PerceptionLM, + title={PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding}, + author={Jang Hyun Cho and Andrea Madotto and Effrosyni Mavroudi and Triantafyllos Afouras and Tushar Nagarajan and Muhammad Maaz and Yale Song and Tengyu Ma and Shuming Hu and Hanoona Rasheed and Peize Sun and Po-Yao Huang and Daniel Bolya and Suyog Jain and Miguel Martin and Huiyu Wang and Nikhila Ravi and Shashank Jain and Temmy Stark and Shane Moon and Babak Damavandi and Vivian Lee and Andrew Westbury and Salman Khan and Philipp Kr\"{a}henb\"{u}hl and Piotr Doll{\'a}r and Lorenzo Torresani and Kristen Grauman and Christoph Feichtenhofer}, + journal={arXiv:2504.13180}, + year={2025} +} +``` diff --git a/perception_models/__pycache__/legrad_pe_audio.cpython-310.pyc b/perception_models/__pycache__/legrad_pe_audio.cpython-310.pyc new file mode 100644 index 0000000000000000000000000000000000000000..b9fb706806082fd72df93d5cf078e2ab2967acd0 Binary files /dev/null and b/perception_models/__pycache__/legrad_pe_audio.cpython-310.pyc differ diff --git a/perception_models/__pycache__/legrad_pe_audio.cpython-313.pyc b/perception_models/__pycache__/legrad_pe_audio.cpython-313.pyc new file mode 100644 index 0000000000000000000000000000000000000000..2a32352f5b0b7ba8672c74dafa3b50a10a0a6607 Binary files /dev/null and b/perception_models/__pycache__/legrad_pe_audio.cpython-313.pyc differ diff --git a/perception_models/__pycache__/legrad_pe_image.cpython-312.pyc b/perception_models/__pycache__/legrad_pe_image.cpython-312.pyc new file mode 100644 index 0000000000000000000000000000000000000000..42c4594a67f6b61e93fdfd364a80eb8abb0f6b68 Binary files /dev/null and b/perception_models/__pycache__/legrad_pe_image.cpython-312.pyc differ diff --git a/perception_models/__pycache__/legrad_pe_image.cpython-313.pyc b/perception_models/__pycache__/legrad_pe_image.cpython-313.pyc new file mode 100644 index 0000000000000000000000000000000000000000..c971797eeaf8575ef77f34a665bb347bfb4b0baf Binary files /dev/null and b/perception_models/__pycache__/legrad_pe_image.cpython-313.pyc differ diff --git a/perception_models/apps/detection/DETA_pe/README.md b/perception_models/apps/detection/DETA_pe/README.md new file mode 100644 index 0000000000000000000000000000000000000000..4ecc3d4001f6b05912a1741b4af7ed48685d67e8 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/README.md @@ -0,0 +1,53 @@ +# SOTA COCO Object Detection with PE + +## Getting started + +Please refer to [INSTALL.md](../INSTALL.md) for installation and dataset preparation instructions. + +Also install [Deformable Attention](models/ops/make.sh) ops. + +## Results and Fine-tuned Models + + + + + + + + + + + + + + + + + +
detectorvision encoderbox
AP		box(TTA)
AP		download
DETAPE spatial G 65.2 66.0 model
+ + +## Training +We apply a four-stage training, Objects365(12ep, 1024pix), Objects365(6ep, 1536pix), COCO(12ep, 1728pix), COCO(3ep, 1824pix) + +``` +sbatch scripts/pretrain_spatial_Gwin384_o365ep12_1024pix_16node.sh + +sbatch scripts/pretrain_continue_spatial_Gwin384_o365ep6_1536pix_16node.sh + +sbatch scripts/finetune_spatial_Gwin384_cocoep12_1728pix_8node.sh + +sbatch scripts/finetune_further_spatial_Gwin384_cocoep3_1824pix_8node.sh + +``` + +## Evaluation +``` +bash scripts/eval_1824pix.sh --resume deta_coco_1824pix.pth +``` + +## Evaluation with TTA (Test-Time Augmentation) +``` +sbatch scripts/eval_tta_slurm_1824pix.sh --resume deta_coco_1824pix.pth +``` +Note: If you get 65.9 AP, it is probably caused by different package versions, trying different hyperparameters like `--quad_scale 0.4` will give 66.0 AP. diff --git a/perception_models/apps/detection/DETA_pe/datasets/__init__.py b/perception_models/apps/detection/DETA_pe/datasets/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..f154383b7114f25835c2b7161a1ab1f361f73305 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/datasets/__init__.py @@ -0,0 +1,37 @@ +# ------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------ +# Modified from DETR (https://github.com/facebookresearch/detr) +# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved +# ------------------------------------------------------------------------ + +import torch.utils.data + +from .coco import build as build_coco +from .objects365 import build as build_objects365 +from .torchvision_datasets import CocoDetection + + +def get_coco_api_from_dataset(dataset): + for _ in range(10): + # if isinstance(dataset, torchvision.datasets.CocoDetection): + # break + if isinstance(dataset, torch.utils.data.Subset): + dataset = dataset.dataset + if isinstance(dataset, CocoDetection): + return dataset.coco + + +def build_dataset(image_set, args): + if args.dataset_file == "objects365": + return build_objects365(image_set, args) + if args.dataset_file == "coco": + return build_coco(image_set, args) + if args.dataset_file == "coco_panoptic": + # to avoid making panopticapi required for coco + from .coco_panoptic import build as build_coco_panoptic + + return build_coco_panoptic(image_set, args) + raise ValueError(f"dataset {args.dataset_file} not supported") diff --git a/perception_models/apps/detection/DETA_pe/datasets/coco.py b/perception_models/apps/detection/DETA_pe/datasets/coco.py new file mode 100644 index 0000000000000000000000000000000000000000..74a55f8f565fbf8875ccf7866389c0a44041ffd6 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/datasets/coco.py @@ -0,0 +1,345 @@ +# ------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------ +# Modified from DETR (https://github.com/facebookresearch/detr) +# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved +# ------------------------------------------------------------------------ + +""" +COCO dataset which returns image_id for evaluation. + +Mostly copy-paste from https://github.com/pytorch/vision/blob/13b35ff/references/detection/coco_utils.py +""" +import random +from pathlib import Path + +import datasets.transforms as T +import torch +import torch.utils.data +import torchvision.transforms.functional as F +from pycocotools import mask as coco_mask +from util.misc import get_local_rank, get_local_size + +from .torchvision_datasets import CocoDetection as TvCocoDetection + + + +class CocoDetection(TvCocoDetection): + def __init__( + self, + img_folder, + ann_file, + transforms, + return_masks, + cache_mode=False, + local_rank=0, + local_size=1, + test_hflip_aug=False, + tta=False, + is_train=False, + lsj_img_size=1824, + ): + super(CocoDetection, self).__init__( + img_folder, + ann_file, + cache_mode=cache_mode, + local_rank=local_rank, + local_size=local_size, + ) + self._transforms = transforms + self.prepare = ConvertCocoPolysToMask(return_masks) + self.test_hflip_aug = test_hflip_aug + self.tta = tta + if lsj_img_size == 1728: # for back-compatibility + self.tta_image_size = [1536, 1152,] + else: + self.tta_image_size = [1728, 1536, 1344,] + + self.is_train = is_train + + def __getitem__(self, idx): + img, target = super(CocoDetection, self).__getitem__(idx) + image_id = self.ids[idx] + target = {"image_id": image_id, "annotations": target} + img, target = self.prepare(img, target) + if self._transforms is not None: + img, target = self._transforms(img, target) + + if self.test_hflip_aug: + flipped_img = torch.flip(img, dims=[-1]) + new_img = torch.cat([img, flipped_img], dim=0) + return new_img, target + + elif self.tta: + tta_images = [img] + flipped_img = torch.flip(img, dims=[-1]) + tta_images.append(flipped_img) + _, height, width = img.shape + max_size_len = height if height >= width else width + for new_max_size in self.tta_image_size: + scale = new_max_size / max_size_len + new_height, new_width = int(scale * height), int(scale * width) + new_img = F.resize(img, size=(new_height, new_width)) + tta_images.append(new_img) + flipped_img = torch.flip(new_img, dims=[-1]) + tta_images.append(flipped_img) + return tta_images, target + else: + return img, target + + +def convert_coco_poly_to_mask(segmentations, height, width): + masks = [] + for polygons in segmentations: + rles = coco_mask.frPyObjects(polygons, height, width) + mask = coco_mask.decode(rles) + if len(mask.shape) < 3: + mask = mask[..., None] + mask = torch.as_tensor(mask, dtype=torch.uint8) + mask = mask.any(dim=2) + masks.append(mask) + if masks: + masks = torch.stack(masks, dim=0) + else: + masks = torch.zeros((0, height, width), dtype=torch.uint8) + return masks + + +class ConvertCocoPolysToMask(object): + def __init__(self, return_masks=False): + self.return_masks = return_masks + + def __call__(self, image, target): + w, h = image.size + + image_id = target["image_id"] + image_id = torch.tensor([image_id]) + + anno = target["annotations"] + + anno = [obj for obj in anno if "iscrowd" not in obj or obj["iscrowd"] == 0] + + boxes = [obj["bbox"] for obj in anno] + # guard against no boxes via resizing + boxes = torch.as_tensor(boxes, dtype=torch.float32).reshape(-1, 4) + boxes[:, 2:] += boxes[:, :2] + boxes[:, 0::2].clamp_(min=0, max=w) + boxes[:, 1::2].clamp_(min=0, max=h) + + classes = [obj["category_id"] for obj in anno] + classes = torch.tensor(classes, dtype=torch.int64) + + if self.return_masks: + segmentations = [obj["segmentation"] for obj in anno] + masks = convert_coco_poly_to_mask(segmentations, h, w) + + keypoints = None + if anno and "keypoints" in anno[0]: + keypoints = [obj["keypoints"] for obj in anno] + keypoints = torch.as_tensor(keypoints, dtype=torch.float32) + num_keypoints = keypoints.shape[0] + if num_keypoints: + keypoints = keypoints.view(num_keypoints, -1, 3) + + keep = (boxes[:, 3] > boxes[:, 1]) & (boxes[:, 2] > boxes[:, 0]) + boxes = boxes[keep] + classes = classes[keep] + if self.return_masks: + masks = masks[keep] + if keypoints is not None: + keypoints = keypoints[keep] + + target = {} + target["boxes"] = boxes + target["labels"] = classes + if self.return_masks: + target["masks"] = masks + target["image_id"] = image_id + if keypoints is not None: + target["keypoints"] = keypoints + + # for conversion to coco api + area = torch.tensor([obj["area"] for obj in anno]) + iscrowd = torch.tensor( + [obj["iscrowd"] if "iscrowd" in obj else 0 for obj in anno] + ) + target["area"] = area[keep] + target["iscrowd"] = iscrowd[keep] + + target["orig_size"] = torch.as_tensor([int(h), int(w)]) + target["size"] = torch.as_tensor([int(h), int(w)]) + + return image, target + + +def make_coco_transforms(image_set, bigger): + + normalize = T.Compose( + [T.ToTensor(), T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])] + ) + + if "train" in image_set: + scales = [480, 512, 544, 576, 608, 640, 672, 704, 736, 768, 800] + if "val" in image_set or "test" in image_set: + scales = [800] + + max_size = 1333 + if bigger: + scales = [int(1.5 * s) for s in scales] + max_size = 2000 + + if image_set == "train": + augmentation_list = [ + T.RandomHorizontalFlip(), + T.RandomSelect( + T.RandomResize(scales, max_size=max_size), + T.Compose( + [ + T.RandomResize([400, 500, 600]), + T.RandomSizeCrop(384, 600), + T.RandomResize(scales, max_size=max_size), + ] + ), + ), + normalize, + ] + + return T.Compose(augmentation_list) + + if image_set == "val": + return T.Compose( + [ + T.RandomResize(scales, max_size=max_size), + normalize, + ] + ) + + raise ValueError(f"unknown {image_set}") + + +def make_coco_transforms_lsj( + image_set, image_size, lsj_img_train_min=480, lsj_strong_aug=False +): + """ + Reference: https://github.com/facebookresearch/detectron2/blob/main/projects/ViTDet/configs/common/coco_loader_lsj.py + + import detectron2.data.transforms as T + from detectron2 import model_zoo + from detectron2.config import LazyCall as L + + # Data using LSJ + image_size = 1024 + dataloader = model_zoo.get_config("common/data/coco.py").dataloader + dataloader.train.mapper.augmentations = [ + L(T.RandomFlip)(horizontal=True), # flip first + L(T.ResizeScale)( + min_scale=0.1, max_scale=2.0, target_height=image_size, target_width=image_size + ), + L(T.FixedSizeCrop)(crop_size=(image_size, image_size), pad=False), + ] + dataloader.train.mapper.image_format = "RGB" + dataloader.train.total_batch_size = 64 + # recompute boxes due to cropping + dataloader.train.mapper.recompute_boxes = True + + dataloader.test.mapper.augmentations = [ + L(T.ResizeShortestEdge)(short_edge_length=image_size, max_size=image_size), + ] + """ + + """ + In our implementation, we simulate lsj data augmentation by: + (1) first the following augmentations + (2) then padding to (image_size, image_size) in collator, see util/misc/collate_fn_lsj.py + """ + normalize = T.Compose( + [T.ToTensor(), T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])] + ) + + if "train" in image_set: + scales = [scale for scale in range(lsj_img_train_min, image_size, 32)] + if "val" in image_set or "test" in image_set or "unlabel" in image_set: + scales = [image_size - 32] + + # max_size = 1333 + # if bigger: + # scales = [int(1.5 * s) for s in scales] + # max_size = 2000 + max_size = image_size - 32 # for some wired bugs + + augmentation_list = [] + if "train" in image_set: + if lsj_strong_aug: + augmentation_list.extend( + [ + T.ColorJitter((0.4, 0.4, 0.4, 0.1), p=0.5), + T.RandomGrayscale(p=0.2), + # T.RandomErasingP05(), + ] + ) + augmentation_list.extend( + [ + T.RandomHorizontalFlip(), + T.RandomSelect( + # similar to (T.ResizeScale)(min_scale=0.1, max_scale=1.0, target_height=image_size, target_width=image_size) and pad + T.RandomResize(scales, max_size=max_size), + # similar to (T.ResizeScale)(min_scale=1.0, max_scale=2.0, target_height=image_size, target_width=image_size) and crop + T.Compose( + [ + T.RandomResize([400, 500, 600]), + T.RandomSizeCrop(384, 600), + T.RandomResize([max_size], max_size=max_size), + ] + ), + ), + normalize, + ] + ) + return T.Compose(augmentation_list) + + if image_set == "val": + return T.Compose( + [ + T.RandomResize(scales, max_size=max_size), + normalize, + ] + ) + + raise ValueError(f"unknown {image_set}") + + +def build(image_set, args): + root = Path(args.coco_path) + assert root.exists(), f"provided COCO path {root} does not exist" + mode = "instances" + PATHS = { + "train": (root / "train2017", root / "annotations" / f"{mode}_train2017.json"), + "val": (root / "val2017", root / "annotations" / f"{mode}_val2017.json"), + } + + img_folder, ann_file = PATHS[image_set] + if args.lsj: + coco_transform = make_coco_transforms_lsj( + image_set, + args.lsj_img_size, + args.lsj_img_train_min, + args.lsj_strong_aug, + ) + else: + coco_transform = make_coco_transforms(image_set, args.bigger) + dataset = CocoDetection( + img_folder, + ann_file, + transforms=coco_transform, + return_masks=args.masks, + cache_mode=args.cache_mode, + local_rank=get_local_rank(), + local_size=get_local_size(), + test_hflip_aug=args.test_hflip_aug, + tta=args.tta, + is_train=("train" in image_set), + lsj_img_size=args.lsj_img_size, + ) + return dataset diff --git a/perception_models/apps/detection/DETA_pe/datasets/coco_eval.py b/perception_models/apps/detection/DETA_pe/datasets/coco_eval.py new file mode 100644 index 0000000000000000000000000000000000000000..9a3ebe7e7d12d07b35d655d1f8f192d44e885cac --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/datasets/coco_eval.py @@ -0,0 +1,265 @@ +# ------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------ +# Modified from DETR (https://github.com/facebookresearch/detr) +# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved +# ------------------------------------------------------------------------ + +""" +COCO evaluator that works in distributed mode. + +Mostly copy-paste from https://github.com/pytorch/vision/blob/edfd5a7/references/detection/coco_eval.py +The difference is that there is less copy-pasting from pycocotools +in the end of the file, as python3 can suppress prints with contextlib +""" +import os +import contextlib +import copy +import numpy as np +import torch + +from pycocotools.cocoeval import COCOeval +from pycocotools.coco import COCO +import pycocotools.mask as mask_util + +from util.misc import all_gather + + +class CocoEvaluator(object): + def __init__(self, coco_gt, iou_types): + assert isinstance(iou_types, (list, tuple)) + coco_gt = copy.deepcopy(coco_gt) + self.coco_gt = coco_gt + + self.iou_types = iou_types + self.coco_eval = {} + for iou_type in iou_types: + self.coco_eval[iou_type] = COCOeval(coco_gt, iouType=iou_type) + + self.img_ids = [] + self.eval_imgs = {k: [] for k in iou_types} + + def update(self, predictions): + img_ids = list(np.unique(list(predictions.keys()))) + self.img_ids.extend(img_ids) + + for iou_type in self.iou_types: + results = self.prepare(predictions, iou_type) + + # suppress pycocotools prints + with open(os.devnull, 'w') as devnull: + with contextlib.redirect_stdout(devnull): + coco_dt = COCO.loadRes(self.coco_gt, results) if results else COCO() + coco_eval = self.coco_eval[iou_type] + + coco_eval.cocoDt = coco_dt + coco_eval.params.imgIds = list(img_ids) + img_ids, eval_imgs = evaluate(coco_eval) + + self.eval_imgs[iou_type].append(eval_imgs) + + def synchronize_between_processes(self): + for iou_type in self.iou_types: + self.eval_imgs[iou_type] = np.concatenate(self.eval_imgs[iou_type], 2) + create_common_coco_eval(self.coco_eval[iou_type], self.img_ids, self.eval_imgs[iou_type]) + + def accumulate(self): + for coco_eval in self.coco_eval.values(): + coco_eval.accumulate() + + def summarize(self): + for iou_type, coco_eval in self.coco_eval.items(): + print("IoU metric: {}".format(iou_type)) + coco_eval.summarize() + + def prepare(self, predictions, iou_type): + if iou_type == "bbox": + return self.prepare_for_coco_detection(predictions) + elif iou_type == "segm": + return self.prepare_for_coco_segmentation(predictions) + elif iou_type == "keypoints": + return self.prepare_for_coco_keypoint(predictions) + else: + raise ValueError("Unknown iou type {}".format(iou_type)) + + def prepare_for_coco_detection(self, predictions): + coco_results = [] + for original_id, prediction in predictions.items(): + if len(prediction) == 0: + continue + + boxes = prediction["boxes"] + boxes = convert_to_xywh(boxes).tolist() + scores = prediction["scores"].tolist() + labels = prediction["labels"].tolist() + + coco_results.extend( + [ + { + "image_id": original_id, + "category_id": labels[k], + "bbox": box, + "score": scores[k], + } + for k, box in enumerate(boxes) + ] + ) + return coco_results + + def prepare_for_coco_segmentation(self, predictions): + coco_results = [] + for original_id, prediction in predictions.items(): + if len(prediction) == 0: + continue + + scores = prediction["scores"] + labels = prediction["labels"] + masks = prediction["masks"] + + masks = masks > 0.5 + + scores = prediction["scores"].tolist() + labels = prediction["labels"].tolist() + + rles = [ + mask_util.encode(np.array(mask[0, :, :, np.newaxis], dtype=np.uint8, order="F"))[0] + for mask in masks + ] + for rle in rles: + rle["counts"] = rle["counts"].decode("utf-8") + + coco_results.extend( + [ + { + "image_id": original_id, + "category_id": labels[k], + "segmentation": rle, + "score": scores[k], + } + for k, rle in enumerate(rles) + ] + ) + return coco_results + + def prepare_for_coco_keypoint(self, predictions): + coco_results = [] + for original_id, prediction in predictions.items(): + if len(prediction) == 0: + continue + + boxes = prediction["boxes"] + boxes = convert_to_xywh(boxes).tolist() + scores = prediction["scores"].tolist() + labels = prediction["labels"].tolist() + keypoints = prediction["keypoints"] + keypoints = keypoints.flatten(start_dim=1).tolist() + + coco_results.extend( + [ + { + "image_id": original_id, + "category_id": labels[k], + 'keypoints': keypoint, + "score": scores[k], + } + for k, keypoint in enumerate(keypoints) + ] + ) + return coco_results + + +def convert_to_xywh(boxes): + xmin, ymin, xmax, ymax = boxes.unbind(1) + return torch.stack((xmin, ymin, xmax - xmin, ymax - ymin), dim=1) + + +def merge(img_ids, eval_imgs): + all_img_ids = all_gather(img_ids) + all_eval_imgs = all_gather(eval_imgs) + + merged_img_ids = [] + for p in all_img_ids: + merged_img_ids.extend(p) + + merged_eval_imgs = [] + for p in all_eval_imgs: + merged_eval_imgs.append(p) + + merged_img_ids = np.array(merged_img_ids) + merged_eval_imgs = np.concatenate(merged_eval_imgs, 2) + + # keep only unique (and in sorted order) images + merged_img_ids, idx = np.unique(merged_img_ids, return_index=True) + merged_eval_imgs = merged_eval_imgs[..., idx] + + return merged_img_ids, merged_eval_imgs + + +def create_common_coco_eval(coco_eval, img_ids, eval_imgs): + img_ids, eval_imgs = merge(img_ids, eval_imgs) + img_ids = list(img_ids) + eval_imgs = list(eval_imgs.flatten()) + + coco_eval.evalImgs = eval_imgs + coco_eval.params.imgIds = img_ids + coco_eval._paramsEval = copy.deepcopy(coco_eval.params) + + +################################################################# +# From pycocotools, just removed the prints and fixed +# a Python3 bug about unicode not defined +################################################################# + + +def evaluate(self): + ''' + Run per image evaluation on given images and store results (a list of dict) in self.evalImgs + :return: None + ''' + # tic = time.time() + # print('Running per image evaluation...') + p = self.params + # add backward compatibility if useSegm is specified in params + if p.useSegm is not None: + p.iouType = 'segm' if p.useSegm == 1 else 'bbox' + print('useSegm (deprecated) is not None. Running {} evaluation'.format(p.iouType)) + # print('Evaluate annotation type *{}*'.format(p.iouType)) + p.imgIds = list(np.unique(p.imgIds)) + if p.useCats: + p.catIds = list(np.unique(p.catIds)) + p.maxDets = sorted(p.maxDets) + self.params = p + + self._prepare() + # loop through images, area range, max detection number + catIds = p.catIds if p.useCats else [-1] + + if p.iouType == 'segm' or p.iouType == 'bbox': + computeIoU = self.computeIoU + elif p.iouType == 'keypoints': + computeIoU = self.computeOks + self.ious = { + (imgId, catId): computeIoU(imgId, catId) + for imgId in p.imgIds + for catId in catIds} + + evaluateImg = self.evaluateImg + maxDet = p.maxDets[-1] + evalImgs = [ + evaluateImg(imgId, catId, areaRng, maxDet) + for catId in catIds + for areaRng in p.areaRng + for imgId in p.imgIds + ] + # this is NOT in the pycocotools code, but could be done outside + evalImgs = np.asarray(evalImgs).reshape(len(catIds), len(p.areaRng), len(p.imgIds)) + self._paramsEval = copy.deepcopy(self.params) + # toc = time.time() + # print('DONE (t={:0.2f}s).'.format(toc-tic)) + return p.imgIds, evalImgs + +################################################################# +# end of straight copy from pycocotools, just removing the prints +################################################################# diff --git a/perception_models/apps/detection/DETA_pe/datasets/coco_panoptic.py b/perception_models/apps/detection/DETA_pe/datasets/coco_panoptic.py new file mode 100644 index 0000000000000000000000000000000000000000..e856e49d84853e5feb78ec0cf002677375cff2bb --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/datasets/coco_panoptic.py @@ -0,0 +1,107 @@ +# ------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------ +# Modified from DETR (https://github.com/facebookresearch/detr) +# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved +# ------------------------------------------------------------------------ + +import json +from pathlib import Path + +import numpy as np +import torch +from PIL import Image + +from panopticapi.utils import rgb2id +from util.box_ops import masks_to_boxes + +from .coco import make_coco_transforms + + +class CocoPanoptic: + def __init__(self, img_folder, ann_folder, ann_file, transforms=None, return_masks=True): + with open(ann_file, 'r') as f: + self.coco = json.load(f) + + # sort 'images' field so that they are aligned with 'annotations' + # i.e., in alphabetical order + self.coco['images'] = sorted(self.coco['images'], key=lambda x: x['id']) + # sanity check + if "annotations" in self.coco: + for img, ann in zip(self.coco['images'], self.coco['annotations']): + assert img['file_name'][:-4] == ann['file_name'][:-4] + + self.img_folder = img_folder + self.ann_folder = ann_folder + self.ann_file = ann_file + self.transforms = transforms + self.return_masks = return_masks + + def __getitem__(self, idx): + ann_info = self.coco['annotations'][idx] if "annotations" in self.coco else self.coco['images'][idx] + img_path = Path(self.img_folder) / ann_info['file_name'].replace('.png', '.jpg') + ann_path = Path(self.ann_folder) / ann_info['file_name'] + + img = Image.open(img_path).convert('RGB') + w, h = img.size + if "segments_info" in ann_info: + masks = np.asarray(Image.open(ann_path), dtype=np.uint32) + masks = rgb2id(masks) + + ids = np.array([ann['id'] for ann in ann_info['segments_info']]) + masks = masks == ids[:, None, None] + + masks = torch.as_tensor(masks, dtype=torch.uint8) + labels = torch.tensor([ann['category_id'] for ann in ann_info['segments_info']], dtype=torch.int64) + + target = {} + target['image_id'] = torch.tensor([ann_info['image_id'] if "image_id" in ann_info else ann_info["id"]]) + if self.return_masks: + target['masks'] = masks + target['labels'] = labels + + target["boxes"] = masks_to_boxes(masks) + + target['size'] = torch.as_tensor([int(h), int(w)]) + target['orig_size'] = torch.as_tensor([int(h), int(w)]) + if "segments_info" in ann_info: + for name in ['iscrowd', 'area']: + target[name] = torch.tensor([ann[name] for ann in ann_info['segments_info']]) + + if self.transforms is not None: + img, target = self.transforms(img, target) + + return img, target + + def __len__(self): + return len(self.coco['images']) + + def get_height_and_width(self, idx): + img_info = self.coco['images'][idx] + height = img_info['height'] + width = img_info['width'] + return height, width + + +def build(image_set, args): + img_folder_root = Path(args.coco_path) + ann_folder_root = Path(args.coco_panoptic_path) + assert img_folder_root.exists(), f'provided COCO path {img_folder_root} does not exist' + assert ann_folder_root.exists(), f'provided COCO path {ann_folder_root} does not exist' + mode = 'panoptic' + PATHS = { + "train": ("train2017", Path("annotations") / f'{mode}_train2017.json'), + "val": ("val2017", Path("annotations") / f'{mode}_val2017.json'), + } + + img_folder, ann_file = PATHS[image_set] + img_folder_path = img_folder_root / img_folder + ann_folder = ann_folder_root / f'{mode}_{img_folder}' + ann_file = ann_folder_root / ann_file + + dataset = CocoPanoptic(img_folder_path, ann_folder, ann_file, + transforms=make_coco_transforms(image_set), return_masks=args.masks) + + return dataset diff --git a/perception_models/apps/detection/DETA_pe/datasets/data_prefetcher.py b/perception_models/apps/detection/DETA_pe/datasets/data_prefetcher.py new file mode 100644 index 0000000000000000000000000000000000000000..7d28d9fdd7f90d2689590c9a126226c068081bd5 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/datasets/data_prefetcher.py @@ -0,0 +1,70 @@ +# ------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------ + +import torch + +def to_cuda(samples, targets, device): + samples = samples.to(device, non_blocking=True) + targets = [{k: v.to(device, non_blocking=True) for k, v in t.items()} for t in targets] + return samples, targets + +class data_prefetcher(): + def __init__(self, loader, device, prefetch=True): + self.loader = iter(loader) + self.prefetch = prefetch + self.device = device + if prefetch: + self.stream = torch.cuda.Stream() + self.preload() + + def preload(self): + try: + self.next_samples, self.next_targets = next(self.loader) + except StopIteration: + self.next_samples = None + self.next_targets = None + return + # if record_stream() doesn't work, another option is to make sure device inputs are created + # on the main stream. + # self.next_input_gpu = torch.empty_like(self.next_input, device='cuda') + # self.next_target_gpu = torch.empty_like(self.next_target, device='cuda') + # Need to make sure the memory allocated for next_* is not still in use by the main stream + # at the time we start copying to next_*: + # self.stream.wait_stream(torch.cuda.current_stream()) + with torch.cuda.stream(self.stream): + self.next_samples, self.next_targets = to_cuda(self.next_samples, self.next_targets, self.device) + # more code for the alternative if record_stream() doesn't work: + # copy_ will record the use of the pinned source tensor in this side stream. + # self.next_input_gpu.copy_(self.next_input, non_blocking=True) + # self.next_target_gpu.copy_(self.next_target, non_blocking=True) + # self.next_input = self.next_input_gpu + # self.next_target = self.next_target_gpu + + # With Amp, it isn't necessary to manually convert data to half. + # if args.fp16: + # self.next_input = self.next_input.half() + # else: + + def next(self): + if self.prefetch: + torch.cuda.current_stream().wait_stream(self.stream) + samples = self.next_samples + targets = self.next_targets + if samples is not None: + samples.record_stream(torch.cuda.current_stream()) + if targets is not None: + for t in targets: + for k, v in t.items(): + v.record_stream(torch.cuda.current_stream()) + self.preload() + else: + try: + samples, targets = next(self.loader) + samples, targets = to_cuda(samples, targets, self.device) + except StopIteration: + samples = None + targets = None + return samples, targets diff --git a/perception_models/apps/detection/DETA_pe/datasets/objects365.py b/perception_models/apps/detection/DETA_pe/datasets/objects365.py new file mode 100644 index 0000000000000000000000000000000000000000..16a1a240a72e288908e3d8f513e8e4e87de7e314 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/datasets/objects365.py @@ -0,0 +1,54 @@ +# ------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------ +# Modified from DETR (https://github.com/facebookresearch/detr) +# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved +# ------------------------------------------------------------------------ + +""" +COCO dataset which returns image_id for evaluation. + +Mostly copy-paste from https://github.com/pytorch/vision/blob/13b35ff/references/detection/coco_utils.py +""" +from pathlib import Path + +import datasets.transforms as T + +import torch +import torch.utils.data +from pycocotools import mask as coco_mask +from util.misc import get_local_rank, get_local_size + +from .coco import CocoDetection, make_coco_transforms, make_coco_transforms_lsj +from .torchvision_datasets import CocoDetection as TvCocoDetection + + +def build(image_set, args): + root = Path(args.coco_path) + assert root.exists(), f"provided Objects365 path {root} does not exist" + mode = "instances" + PATHS = { + "train": ( + root / "train", + root / "annotations" / "zhiyuan_objv2_train_fixmiss.json", + ), + "val": (root / "val", root / "annotations" / "zhiyuan_objv2_val.json"), + } + + img_folder, ann_file = PATHS[image_set] + if args.lsj: + coco_transform = make_coco_transforms_lsj(image_set, args.lsj_img_size) + else: + coco_transform = make_coco_transforms(image_set, args.bigger) + dataset = CocoDetection( + img_folder, + ann_file, + transforms=coco_transform, + return_masks=args.masks, + cache_mode=args.cache_mode, + local_rank=get_local_rank(), + local_size=get_local_size(), + ) + return dataset diff --git a/perception_models/apps/detection/DETA_pe/datasets/panoptic_eval.py b/perception_models/apps/detection/DETA_pe/datasets/panoptic_eval.py new file mode 100644 index 0000000000000000000000000000000000000000..0dabffdb584210af1836d9ee7e5b12532926e9b0 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/datasets/panoptic_eval.py @@ -0,0 +1,52 @@ +# ------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------ +# Modified from DETR (https://github.com/facebookresearch/detr) +# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved +# ------------------------------------------------------------------------ + +import json +import os + +import util.misc as utils + +try: + from panopticapi.evaluation import pq_compute +except ImportError: + pass + + +class PanopticEvaluator(object): + def __init__(self, ann_file, ann_folder, output_dir="panoptic_eval"): + self.gt_json = ann_file + self.gt_folder = ann_folder + if utils.is_main_process(): + if not os.path.exists(output_dir): + os.mkdir(output_dir) + self.output_dir = output_dir + self.predictions = [] + + def update(self, predictions): + for p in predictions: + with open(os.path.join(self.output_dir, p["file_name"]), "wb") as f: + f.write(p.pop("png_string")) + + self.predictions += predictions + + def synchronize_between_processes(self): + all_predictions = utils.all_gather(self.predictions) + merged_predictions = [] + for p in all_predictions: + merged_predictions += p + self.predictions = merged_predictions + + def summarize(self): + if utils.is_main_process(): + json_data = {"annotations": self.predictions} + predictions_json = os.path.join(self.output_dir, "predictions.json") + with open(predictions_json, "w") as f: + f.write(json.dumps(json_data)) + return pq_compute(self.gt_json, predictions_json, gt_folder=self.gt_folder, pred_folder=self.output_dir) + return None diff --git a/perception_models/apps/detection/DETA_pe/datasets/samplers.py b/perception_models/apps/detection/DETA_pe/datasets/samplers.py new file mode 100644 index 0000000000000000000000000000000000000000..5d3da7142d2523501aeca9d2d1cb6c41955a5465 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/datasets/samplers.py @@ -0,0 +1,348 @@ +# ------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------ +# Modified from codes in torch.utils.data.distributed +# ------------------------------------------------------------------------ + +import json +import math +import os +from collections import defaultdict + +import torch +import torch.distributed as dist + +from fvcore.common.timer import Timer +from lvis import LVIS +from torch.utils.data.sampler import Sampler + + +def load_dataset_dicts(json_file): + timer = Timer() + lvis_api = LVIS(json_file) + if timer.seconds() > 1: + print("Loading {} takes {:.2f} seconds.".format(json_file, timer.seconds())) + + img_ids = sorted(lvis_api.imgs.keys()) + imgs = lvis_api.load_imgs(img_ids) + anns = [lvis_api.img_ann_map[img_id] for img_id in img_ids] + + imgs_anns = list(zip(imgs, anns)) + print( + "Loaded {} images in the LVIS format from {}".format(len(imgs_anns), json_file) + ) + dataset_dicts = [] + + for img_dict, anno_dict_list in imgs_anns: + record = {} + image_id = record["image_id"] = img_dict["id"] + objs = [] + for anno in anno_dict_list: + # Check that the image_id in this annotation is the same as + # the image_id we're looking at. + # This fails only when the data parsing logic or the annotation file is buggy. + assert anno["image_id"] == image_id + obj = {} + # Convert 1-indexed to 0-indexed + obj["category_id"] = anno["category_id"] - 1 + + objs.append(obj) + record["annotations"] = objs + dataset_dicts.append(record) + + return dataset_dicts + + +def repeat_factors_from_category_frequency(dataset_dicts, repeat_thresh, sqrt=True): + # 1. For each category c, compute the fraction of images that contain it: f(c) + category_freq = defaultdict(int) + for dataset_dict in dataset_dicts: # For each image (without repeats) + cat_ids = {ann["category_id"] for ann in dataset_dict["annotations"]} + for cat_id in cat_ids: + category_freq[cat_id] += 1 + num_images = len(dataset_dicts) + for k, v in category_freq.items(): + category_freq[k] = v / num_images + + # 2. For each category c, compute the category-level repeat factor: + # r(c) = max(1, sqrt(t / f(c))) + category_rep = { + cat_id: max( + 1.0, + ( + math.sqrt(repeat_thresh / cat_freq) + if sqrt + else (repeat_thresh / cat_freq) + ), + ) + for cat_id, cat_freq in category_freq.items() + } + for cat_id in sorted(category_rep.keys()): + print( + f"Cat ID {cat_id}: freq={category_freq[cat_id]:.2f}, rep={category_rep[cat_id]:.2f}" + ) + + # 3. For each image I, compute the image-level repeat factor: + # r(I) = max_{c in I} r(c) + rep_factors = [] + for dataset_dict in dataset_dicts: + cat_ids = {ann["category_id"] for ann in dataset_dict["annotations"]} + rep_factor = max({category_rep[cat_id] for cat_id in cat_ids}, default=1.0) + rep_factors.append(rep_factor) + + return torch.tensor(rep_factors, dtype=torch.float32) + + +class RepeatFactorTrainingSampler(Sampler): + def __init__( + self, + dataset, + num_replicas=None, + rank=None, + local_rank=None, + local_size=None, + shuffle=True, + ): + if num_replicas is None: + if not dist.is_available(): + raise RuntimeError("Requires distributed package to be available") + num_replicas = dist.get_world_size() + if rank is None: + if not dist.is_available(): + raise RuntimeError("Requires distributed package to be available") + rank = dist.get_rank() + self.dataset = dataset + self.num_replicas = num_replicas + self.rank = rank + self.epoch = 0 + self.num_samples = int(math.ceil(len(self.dataset) * 1.0 / self.num_replicas)) + self.total_size = self.num_samples * self.num_replicas + self.shuffle = shuffle + + json_file = ( + "/checkpoint/onevision/peizesun/public_data/d2_data/lvis/lvis_v1_train.json" + ) + dataset_dicts = load_dataset_dicts(json_file) + repeat_factors = repeat_factors_from_category_frequency( + dataset_dicts, repeat_thresh=0.001 + ) + # Split into whole number (_int_part) and fractional (_frac_part) parts. + self._int_part = torch.trunc(repeat_factors) + self._frac_part = repeat_factors - self._int_part + + def _get_epoch_indices(self, generator): + """ + Create a list of dataset indices (with repeats) to use for one epoch. + + Args: + generator (torch.Generator): pseudo random number generator used for + stochastic rounding. + + Returns: + torch.Tensor: list of dataset indices to use in one epoch. Each index + is repeated based on its calculated repeat factor. + """ + # Since repeat factors are fractional, we use stochastic rounding so + # that the target repeat factor is achieved in expectation over the + # course of training + rands = torch.rand(len(self._frac_part), generator=generator) + rep_factors = self._int_part + (rands < self._frac_part).float() + # Construct a list of indices in which we repeat images as specified + indices = [] + for dataset_index, rep_factor in enumerate(rep_factors): + indices.extend([dataset_index] * int(rep_factor.item())) + return torch.tensor(indices, dtype=torch.int64) + + def __iter__(self): + if self.shuffle: + g = torch.Generator() + g.manual_seed(self.epoch) + # Sample indices with repeats determined by stochastic rounding; each + # "epoch" may have a slightly different size due to the rounding. + rfs_indices = self._get_epoch_indices(g) + # deterministically shuffle based on epoch + randperm = torch.randperm(len(rfs_indices), generator=g) + indices = rfs_indices[randperm].tolist() + else: + g = torch.Generator() + g.manual_seed(0) + # Sample indices with repeats determined by stochastic rounding; each + # "epoch" may have a slightly different size due to the rounding. + rfs_indices = self._get_epoch_indices(g) + indices = rfs_indices.tolist() + + # add extra samples to make it evenly divisible + if self.total_size > len(indices): + indices += indices[: (self.total_size - len(indices))] + assert len(indices) == self.total_size + # subsample + offset = self.num_samples * self.rank + indices = indices[offset : offset + self.num_samples] + assert len(indices) == self.num_samples + + return iter(indices) + else: + self.num_samples = int(math.ceil(len(indices) * 1.0 / self.num_replicas)) + self.total_size = self.num_samples * self.num_replicas + indices += indices[: (self.total_size - len(indices))] + assert len(indices) == self.total_size + # subsample + offset = self.num_samples * self.rank + indices = indices[offset : offset + self.num_samples] + assert len(indices) == self.num_samples + + return iter(indices) + + def __len__(self): + return self.num_samples + + def set_epoch(self, epoch): + self.epoch = epoch + + +class DistributedSampler(Sampler): + """Sampler that restricts data loading to a subset of the dataset. + It is especially useful in conjunction with + :class:`torch.nn.parallel.DistributedDataParallel`. In such case, each + process can pass a DistributedSampler instance as a DataLoader sampler, + and load a subset of the original dataset that is exclusive to it. + .. note:: + Dataset is assumed to be of constant size. + Arguments: + dataset: Dataset used for sampling. + num_replicas (optional): Number of processes participating in + distributed training. + rank (optional): Rank of the current process within num_replicas. + """ + + def __init__( + self, + dataset, + num_replicas=None, + rank=None, + local_rank=None, + local_size=None, + shuffle=True, + ): + if num_replicas is None: + if not dist.is_available(): + raise RuntimeError("Requires distributed package to be available") + num_replicas = dist.get_world_size() + if rank is None: + if not dist.is_available(): + raise RuntimeError("Requires distributed package to be available") + rank = dist.get_rank() + self.dataset = dataset + self.num_replicas = num_replicas + self.rank = rank + self.epoch = 0 + self.num_samples = int(math.ceil(len(self.dataset) * 1.0 / self.num_replicas)) + self.total_size = self.num_samples * self.num_replicas + self.shuffle = shuffle + + def __iter__(self): + if self.shuffle: + # deterministically shuffle based on epoch + g = torch.Generator() + g.manual_seed(self.epoch) + indices = torch.randperm(len(self.dataset), generator=g).tolist() + else: + indices = torch.arange(len(self.dataset)).tolist() + + # add extra samples to make it evenly divisible + indices += indices[: (self.total_size - len(indices))] + assert len(indices) == self.total_size + + # subsample + offset = self.num_samples * self.rank + indices = indices[offset : offset + self.num_samples] + assert len(indices) == self.num_samples + + return iter(indices) + + def __len__(self): + return self.num_samples + + def set_epoch(self, epoch): + self.epoch = epoch + + +class NodeDistributedSampler(Sampler): + """Sampler that restricts data loading to a subset of the dataset. + It is especially useful in conjunction with + :class:`torch.nn.parallel.DistributedDataParallel`. In such case, each + process can pass a DistributedSampler instance as a DataLoader sampler, + and load a subset of the original dataset that is exclusive to it. + .. note:: + Dataset is assumed to be of constant size. + Arguments: + dataset: Dataset used for sampling. + num_replicas (optional): Number of processes participating in + distributed training. + rank (optional): Rank of the current process within num_replicas. + """ + + def __init__( + self, + dataset, + num_replicas=None, + rank=None, + local_rank=None, + local_size=None, + shuffle=True, + ): + if num_replicas is None: + if not dist.is_available(): + raise RuntimeError("Requires distributed package to be available") + num_replicas = dist.get_world_size() + if rank is None: + if not dist.is_available(): + raise RuntimeError("Requires distributed package to be available") + rank = dist.get_rank() + if local_rank is None: + local_rank = int(os.environ.get("LOCAL_RANK", 0)) + if local_size is None: + local_size = int(os.environ.get("LOCAL_SIZE", 1)) + self.dataset = dataset + self.shuffle = shuffle + self.num_replicas = num_replicas + self.num_parts = local_size + self.rank = rank + self.local_rank = local_rank + self.epoch = 0 + self.num_samples = int(math.ceil(len(self.dataset) * 1.0 / self.num_replicas)) + self.total_size = self.num_samples * self.num_replicas + + self.total_size_parts = self.num_samples * self.num_replicas // self.num_parts + + def __iter__(self): + if self.shuffle: + # deterministically shuffle based on epoch + g = torch.Generator() + g.manual_seed(self.epoch) + indices = torch.randperm(len(self.dataset), generator=g).tolist() + else: + indices = torch.arange(len(self.dataset)).tolist() + indices = [i for i in indices if i % self.num_parts == self.local_rank] + + # add extra samples to make it evenly divisible + indices += indices[: (self.total_size_parts - len(indices))] + assert len(indices) == self.total_size_parts + + # subsample + indices = indices[ + self.rank + // self.num_parts : self.total_size_parts : self.num_replicas + // self.num_parts + ] + assert len(indices) == self.num_samples + + return iter(indices) + + def __len__(self): + return self.num_samples + + def set_epoch(self, epoch): + self.epoch = epoch diff --git a/perception_models/apps/detection/DETA_pe/datasets/torchvision_datasets/__init__.py b/perception_models/apps/detection/DETA_pe/datasets/torchvision_datasets/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..162303c4cea0a2c0c6ecd65143c28b1752cae49a --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/datasets/torchvision_datasets/__init__.py @@ -0,0 +1,7 @@ +# ------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------ + +from .coco import CocoDetection diff --git a/perception_models/apps/detection/DETA_pe/datasets/torchvision_datasets/coco.py b/perception_models/apps/detection/DETA_pe/datasets/torchvision_datasets/coco.py new file mode 100644 index 0000000000000000000000000000000000000000..45b5f52fa9945145dbd48630bac2c2fd42d672b7 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/datasets/torchvision_datasets/coco.py @@ -0,0 +1,84 @@ +# ------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------ +# Modified from torchvision +# ------------------------------------------------------------------------ + +""" +Copy-Paste from torchvision, but add utility of caching images on memory +""" +from torchvision.datasets.vision import VisionDataset +from PIL import Image +import os +import os.path +import tqdm +from io import BytesIO + + +class CocoDetection(VisionDataset): + """`MS Coco Detection `_ Dataset. + Args: + root (string): Root directory where images are downloaded to. + annFile (string): Path to json annotation file. + transform (callable, optional): A function/transform that takes in an PIL image + and returns a transformed version. E.g, ``transforms.ToTensor`` + target_transform (callable, optional): A function/transform that takes in the + target and transforms it. + transforms (callable, optional): A function/transform that takes input sample and its target as entry + and returns a transformed version. + """ + + def __init__(self, root, annFile, transform=None, target_transform=None, transforms=None, + cache_mode=False, local_rank=0, local_size=1): + super(CocoDetection, self).__init__(root, transforms, transform, target_transform) + from pycocotools.coco import COCO + self.coco = COCO(annFile) + self.ids = list(sorted(self.coco.imgs.keys())) + self.cache_mode = cache_mode + self.local_rank = local_rank + self.local_size = local_size + if cache_mode: + self.cache = {} + self.cache_images() + + def cache_images(self): + self.cache = {} + for index, img_id in zip(tqdm.trange(len(self.ids)), self.ids): + if index % self.local_size != self.local_rank: + continue + path = self.coco.loadImgs(img_id)[0]['file_name'] + with open(os.path.join(self.root, path), 'rb') as f: + self.cache[path] = f.read() + + def get_image(self, path): + if self.cache_mode: + if path not in self.cache.keys(): + with open(os.path.join(self.root, path), 'rb') as f: + self.cache[path] = f.read() + return Image.open(BytesIO(self.cache[path])).convert('RGB') + return Image.open(os.path.join(self.root, path)).convert('RGB') + + def __getitem__(self, index): + """ + Args: + index (int): Index + Returns: + tuple: Tuple (image, target). target is the object returned by ``coco.loadAnns``. + """ + coco = self.coco + img_id = self.ids[index] + ann_ids = coco.getAnnIds(imgIds=img_id) + target = coco.loadAnns(ann_ids) + + path = coco.loadImgs(img_id)[0]['file_name'] + + img = self.get_image(path) + if self.transforms is not None: + img, target = self.transforms(img, target) + + return img, target + + def __len__(self): + return len(self.ids) diff --git a/perception_models/apps/detection/DETA_pe/datasets/transforms.py b/perception_models/apps/detection/DETA_pe/datasets/transforms.py new file mode 100644 index 0000000000000000000000000000000000000000..99c2798e88d1a4e9c31da4cd1135cce41e8efea8 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/datasets/transforms.py @@ -0,0 +1,327 @@ +# ------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------ +# Modified from DETR (https://github.com/facebookresearch/detr) +# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved +# ------------------------------------------------------------------------ + +""" +Transforms and data augmentation for both image + bbox. +""" +import random + +import PIL +import torch +import torchvision.transforms as T +import torchvision.transforms.functional as F + +from util.box_ops import box_xyxy_to_cxcywh +from util.misc import interpolate + + +def crop(image, target, region): + cropped_image = F.crop(image, *region) + + target = target.copy() + i, j, h, w = region + + # should we do something wrt the original size? + target["size"] = torch.tensor([h, w]) + + fields = ["labels", "area", "iscrowd"] + + if "boxes" in target: + boxes = target["boxes"] + max_size = torch.as_tensor([w, h], dtype=torch.float32) + cropped_boxes = boxes - torch.as_tensor([j, i, j, i]) + cropped_boxes = torch.min(cropped_boxes.reshape(-1, 2, 2), max_size) + cropped_boxes = cropped_boxes.clamp(min=0) + area = (cropped_boxes[:, 1, :] - cropped_boxes[:, 0, :]).prod(dim=1) + target["boxes"] = cropped_boxes.reshape(-1, 4) + target["area"] = area + fields.append("boxes") + + if "masks" in target: + # FIXME should we update the area here if there are no boxes? + target["masks"] = target["masks"][:, i : i + h, j : j + w] + fields.append("masks") + + # remove elements for which the boxes or masks that have zero area + if "boxes" in target or "masks" in target: + # favor boxes selection when defining which elements to keep + # this is compatible with previous implementation + if "boxes" in target: + cropped_boxes = target["boxes"].reshape(-1, 2, 2) + keep = torch.all(cropped_boxes[:, 1, :] > cropped_boxes[:, 0, :], dim=1) + else: + keep = target["masks"].flatten(1).any(1) + + for field in fields: + target[field] = target[field][keep] + + return cropped_image, target + + +def hflip(image, target): + flipped_image = F.hflip(image) + + w, h = image.size + + target = target.copy() + if "boxes" in target: + boxes = target["boxes"] + boxes = boxes[:, [2, 1, 0, 3]] * torch.as_tensor( + [-1, 1, -1, 1] + ) + torch.as_tensor([w, 0, w, 0]) + target["boxes"] = boxes + + if "masks" in target: + target["masks"] = target["masks"].flip(-1) + + return flipped_image, target + + +def resize(image, target, size, max_size=None): + # size can be min_size (scalar) or (w, h) tuple + + def get_size_with_aspect_ratio(image_size, size, max_size=None): + w, h = image_size + if max_size is not None: + min_original_size = float(min((w, h))) + max_original_size = float(max((w, h))) + if max_original_size / min_original_size * size > max_size: + size = int(round(max_size * min_original_size / max_original_size)) + + if (w <= h and w == size) or (h <= w and h == size): + return (h, w) + if w < h: + ow = size + oh = int(size * h / w) + else: + oh = size + ow = int(size * w / h) + return (oh, ow) + + def get_size(image_size, size, max_size=None): + if isinstance(size, (list, tuple)): + return size[::-1] + else: + return get_size_with_aspect_ratio(image_size, size, max_size) + + size = get_size(image.size, size, max_size) + rescaled_image = F.resize(image, size) + + if target is None: + return rescaled_image, None + + ratios = tuple( + float(s) / float(s_orig) for s, s_orig in zip(rescaled_image.size, image.size) + ) + ratio_width, ratio_height = ratios + + target = target.copy() + if "boxes" in target: + boxes = target["boxes"] + scaled_boxes = boxes * torch.as_tensor( + [ratio_width, ratio_height, ratio_width, ratio_height] + ) + target["boxes"] = scaled_boxes + + if "area" in target: + area = target["area"] + scaled_area = area * (ratio_width * ratio_height) + target["area"] = scaled_area + + h, w = size + target["size"] = torch.tensor([h, w]) + + if "masks" in target: + target["masks"] = ( + interpolate(target["masks"][:, None].float(), size, mode="nearest")[:, 0] + > 0.5 + ) + + return rescaled_image, target + + +def pad(image, target, padding): + # assumes that we only pad on the bottom right corners + padded_image = F.pad(image, (0, 0, padding[0], padding[1])) + if target is None: + return padded_image, None + target = target.copy() + # should we do something wrt the original size? + target["size"] = torch.tensor(padded_image[::-1]) + if "masks" in target: + target["masks"] = torch.nn.functional.pad( + target["masks"], (0, padding[0], 0, padding[1]) + ) + return padded_image, target + + +class RandomCrop(object): + def __init__(self, size): + self.size = size + + def __call__(self, img, target): + region = T.RandomCrop.get_params(img, self.size) + return crop(img, target, region) + + +class RandomSizeCrop(object): + def __init__(self, min_size: int, max_size: int): + self.min_size = min_size + self.max_size = max_size + + def __call__(self, img: PIL.Image.Image, target: dict): + w = random.randint(self.min_size, min(img.width, self.max_size)) + h = random.randint(self.min_size, min(img.height, self.max_size)) + region = T.RandomCrop.get_params(img, [h, w]) + return crop(img, target, region) + + +class CenterCrop(object): + def __init__(self, size): + self.size = size + + def __call__(self, img, target): + image_width, image_height = img.size + crop_height, crop_width = self.size + crop_top = int(round((image_height - crop_height) / 2.0)) + crop_left = int(round((image_width - crop_width) / 2.0)) + return crop(img, target, (crop_top, crop_left, crop_height, crop_width)) + + +class RandomHorizontalFlip(object): + def __init__(self, p=0.5): + self.p = p + + def __call__(self, img, target): + if random.random() < self.p: + return hflip(img, target) + return img, target + + +class RandomResize(object): + def __init__(self, sizes, max_size=None): + assert isinstance(sizes, (list, tuple)) + self.sizes = sizes + self.max_size = max_size + + def __call__(self, img, target=None): + size = random.choice(self.sizes) + return resize(img, target, size, self.max_size) + + +class RandomPad(object): + def __init__(self, max_pad): + self.max_pad = max_pad + + def __call__(self, img, target): + pad_x = random.randint(0, self.max_pad) + pad_y = random.randint(0, self.max_pad) + return pad(img, target, (pad_x, pad_y)) + + +class RandomSelect(object): + """ + Randomly selects between transforms1 and transforms2, + with probability p for transforms1 and (1 - p) for transforms2 + """ + + def __init__(self, transforms1, transforms2, p=0.5): + self.transforms1 = transforms1 + self.transforms2 = transforms2 + self.p = p + + def __call__(self, img, target): + if random.random() < self.p: + return self.transforms1(img, target) + return self.transforms2(img, target) + + +class ToTensor(object): + def __call__(self, img, target): + return F.to_tensor(img), target + + +class RandomErasingP05(object): + def __init__(self): + self.eraser = T.Compose( + [ + T.ToTensor(), + T.RandomErasing( + p=0.5, scale=(0.02, 0.2), ratio=(0.1, 6), value="random" + ), + T.ToPILImage(), + ] + ) + + def __call__(self, img, target): + return self.eraser(img), target + + +class RandomErasing(object): + def __init__(self, *args, **kwargs): + self.eraser = T.RandomErasing(*args, **kwargs) + + def __call__(self, img, target): + return self.eraser(img), target + + +class ColorJitter(object): + def __init__(self, jitter=(0.2, 0.2, 0.2, 0.1), p=0.5): + self.color_jitter = T.ColorJitter(*jitter) + self.p = p + + def __call__(self, img, target): + if random.random() < self.p: + return self.color_jitter(img), target + return img, target + + +class RandomGrayscale(object): + def __init__(self, p=0.5): + self.random_gray = T.RandomGrayscale(p=p) + + def __call__(self, img, target): + return self.random_gray(img), target + + +class Normalize(object): + def __init__(self, mean, std): + self.mean = mean + self.std = std + + def __call__(self, image, target=None): + image = F.normalize(image, mean=self.mean, std=self.std) + if target is None: + return image, None + target = target.copy() + h, w = image.shape[-2:] + if "boxes" in target: + boxes = target["boxes"] + boxes = box_xyxy_to_cxcywh(boxes) + boxes = boxes / torch.tensor([w, h, w, h], dtype=torch.float32) + target["boxes"] = boxes + return image, target + + +class Compose(object): + def __init__(self, transforms): + self.transforms = transforms + + def __call__(self, image, target): + for t in self.transforms: + image, target = t(image, target) + return image, target + + def __repr__(self): + format_string = self.__class__.__name__ + "(" + for t in self.transforms: + format_string += "\n" + format_string += " {0}".format(t) + format_string += "\n)" + return format_string diff --git a/perception_models/apps/detection/DETA_pe/engine.py b/perception_models/apps/detection/DETA_pe/engine.py new file mode 100644 index 0000000000000000000000000000000000000000..385b27df4794b138b75c3832e6262c9fe0f79bb8 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/engine.py @@ -0,0 +1,303 @@ +# ------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------ +# Modified from DETR (https://github.com/facebookresearch/detr) +# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved +# ------------------------------------------------------------------------ + +""" +Train and eval functions used in main.py +""" +import math +import os +import sys +from typing import Iterable + +import torch +import util.misc as utils +from datasets.coco_eval import CocoEvaluator, convert_to_xywh +from datasets.data_prefetcher import data_prefetcher +from datasets.panoptic_eval import PanopticEvaluator +from util.ema import requires_grad, update_ema +from util.misc import NestedTensor + + +def train_one_epoch( + model: torch.nn.Module, + criterion: torch.nn.Module, + data_loader: Iterable, + optimizer: torch.optim.Optimizer, + device: torch.device, + epoch: int, + max_norm: float = 0, + ema: torch.nn.Module = None, + ema_decay: float = 0.999, +): + model.train() + criterion.train() + metric_logger = utils.MetricLogger(delimiter=" ") + metric_logger.add_meter("lr", utils.SmoothedValue(window_size=1, fmt="{value:.6f}")) + metric_logger.add_meter( + "class_error", utils.SmoothedValue(window_size=1, fmt="{value:.2f}") + ) + metric_logger.add_meter( + "grad_norm", utils.SmoothedValue(window_size=1, fmt="{value:.2f}") + ) + header = "Epoch: [{}]".format(epoch) + print_freq = 10 + + prefetcher = data_prefetcher(data_loader, device, prefetch=True) + samples, targets = prefetcher.next() + + # for samples, targets in metric_logger.log_every(data_loader, print_freq, header): + for _ in metric_logger.log_every(range(len(data_loader)), print_freq, header): + outputs = model(samples) + loss_dict = criterion(outputs, targets) + weight_dict = criterion.weight_dict + losses = sum( + loss_dict[k] * weight_dict[k] for k in loss_dict.keys() if k in weight_dict + ) + + # reduce losses over all GPUs for logging purposes + loss_dict_reduced = utils.reduce_dict(loss_dict) + loss_dict_reduced_unscaled = { + f"{k}_unscaled": v for k, v in loss_dict_reduced.items() + } + loss_dict_reduced_scaled = { + k: v * weight_dict[k] + for k, v in loss_dict_reduced.items() + if k in weight_dict + } + losses_reduced_scaled = sum(loss_dict_reduced_scaled.values()) + + loss_value = losses_reduced_scaled.item() + + if not math.isfinite(loss_value): + print("Loss is {}, stopping training".format(loss_value)) + print(loss_dict_reduced) + sys.exit(1) + + optimizer.zero_grad() + losses.backward() + if max_norm > 0: + grad_total_norm = torch.nn.utils.clip_grad_norm_( + model.parameters(), max_norm + ) + else: + grad_total_norm = utils.get_total_grad_norm(model.parameters(), max_norm) + optimizer.step() + + if ema is not None: + update_ema(ema, model.module, ema_decay) + # torch.cuda.empty_cache() + + metric_logger.update( + loss=loss_value, **loss_dict_reduced_scaled, **loss_dict_reduced_unscaled + ) + metric_logger.update(class_error=loss_dict_reduced["class_error"]) + metric_logger.update(lr=optimizer.param_groups[0]["lr"]) + metric_logger.update(grad_norm=grad_total_norm) + + samples, targets = prefetcher.next() + # gather the stats from all processes + metric_logger.synchronize_between_processes() + print("Averaged stats:", metric_logger) + return {k: meter.global_avg for k, meter in metric_logger.meters.items()} + + +@torch.no_grad() +def evaluate( + model_no_ema, + criterion, + postprocessors, + data_loader, + base_ds, + device, + output_dir, + test_hflip_aug, + tta, + soft_nms, + ema=None, + save_result=False, + save_result_dir="", + soft_nms_method="quad", + nms_thresh=0.7, + quad_scale=0.5, + lsj_img_size=1824, +): + model = model_no_ema if ema is None else ema + model.eval() + criterion.eval() + + metric_logger = utils.MetricLogger(delimiter=" ") + metric_logger.add_meter( + "class_error", utils.SmoothedValue(window_size=1, fmt="{value:.2f}") + ) + header = "Test:" + + iou_types = tuple(k for k in ("segm", "bbox") if k in postprocessors.keys()) + coco_evaluator = CocoEvaluator(base_ds, iou_types) + # coco_evaluator.coco_eval[iou_types[0]].params.iouThrs = [0, 0.1, 0.5, 0.75] + + panoptic_evaluator = None + if "panoptic" in postprocessors.keys(): + panoptic_evaluator = PanopticEvaluator( + data_loader.dataset.ann_file, + data_loader.dataset.ann_folder, + output_dir=os.path.join(output_dir, "panoptic_eval"), + ) + + prediction_list = [] + for samples, targets in metric_logger.log_every(data_loader, 10, header): + samples = samples.to(device) + targets = [{k: v.to(device) for k, v in t.items()} for t in targets] + + if test_hflip_aug: + assert ( + samples.tensors.shape[0] == 1 + ), "test_hflip_aug only supports batch size 1" + assert ( + samples.tensors.shape[1] == 6 + ), "test_hflip_aug requires two images in a batch" + first_samples = NestedTensor(samples.tensors[:, :3], samples.mask) + outputs = model(first_samples) + flipped_samples = NestedTensor(samples.tensors[:, 3:], samples.mask) + flipped_outputs = model(flipped_samples) + else: + outputs = model(samples) + loss_dict = criterion(outputs, targets) + weight_dict = criterion.weight_dict + + # reduce losses over all GPUs for logging purposes + loss_dict_reduced = utils.reduce_dict(loss_dict) + loss_dict_reduced_scaled = { + k: v * weight_dict[k] + for k, v in loss_dict_reduced.items() + if k in weight_dict + } + loss_dict_reduced_unscaled = { + f"{k}_unscaled": v for k, v in loss_dict_reduced.items() + } + metric_logger.update( + loss=sum(loss_dict_reduced_scaled.values()), + **loss_dict_reduced_scaled, + **loss_dict_reduced_unscaled, + ) + metric_logger.update(class_error=loss_dict_reduced["class_error"]) + + orig_target_sizes = torch.stack([t["orig_size"] for t in targets], dim=0) + if test_hflip_aug: + new_outputs = {} + pred_logits = outputs["pred_logits"] + pred_boxes = outputs["pred_boxes"] + + flipped_pred_logits = flipped_outputs["pred_logits"] + flipped_pred_boxes = flipped_outputs["pred_boxes"] + + reflipped_pred_boxes = flipped_pred_boxes[ + :, :, [0, 1, 2, 3] + ] * torch.as_tensor([-1, 1, 1, 1]).to( + flipped_pred_boxes.device + ) + torch.as_tensor( + [1, 0, 0, 0] + ).to( + flipped_pred_boxes.device + ) + + new_pred_logits = torch.cat([pred_logits, flipped_pred_logits], dim=1) + new_pred_boxes = torch.cat([pred_boxes, reflipped_pred_boxes], dim=1) + + new_outputs["pred_logits"] = new_pred_logits + new_outputs["pred_boxes"] = new_pred_boxes + results = postprocessors["bbox"]( + new_outputs, + orig_target_sizes, + soft_nms=soft_nms, + method=soft_nms_method, + nms_thresh=nms_thresh, + quad_scale=quad_scale, + ) + else: + results = postprocessors["bbox"]( + outputs, + orig_target_sizes, + soft_nms=soft_nms, + method=soft_nms_method, + nms_thresh=nms_thresh, + quad_scale=quad_scale, + ) + if "segm" in postprocessors.keys(): + target_sizes = torch.stack([t["size"] for t in targets], dim=0) + results = postprocessors["segm"]( + results, outputs, orig_target_sizes, target_sizes + ) + res = { + target["image_id"].item(): output + for target, output in zip(targets, results) + } + if coco_evaluator is not None: + coco_evaluator.update(res) + + if panoptic_evaluator is not None: + res_pano = postprocessors["panoptic"]( + outputs, target_sizes, orig_target_sizes + ) + for i, target in enumerate(targets): + image_id = target["image_id"].item() + file_name = f"{image_id:012d}.png" + res_pano[i]["image_id"] = image_id + res_pano[i]["file_name"] = file_name + + panoptic_evaluator.update(res_pano) + + for target, output in zip(targets, results): + res_cpu = { + target["image_id"].item(): { + "boxes": output["boxes"].cpu(), + "labels": output["labels"].cpu(), + "scores": output["scores"].cpu(), + } + } + prediction_list.append(res_cpu) + + # gather the stats from all processes + metric_logger.synchronize_between_processes() + print("Averaged stats:", metric_logger) + + if save_result: + + from torch import distributed as dist + + os.makedirs(save_result_dir, exist_ok=True) + rank = dist.get_rank() + torch.save( + prediction_list, + os.path.join(save_result_dir, f"val2017_prediction_{rank}.pth"), + ) + + if coco_evaluator is not None: + coco_evaluator.synchronize_between_processes() + if panoptic_evaluator is not None: + panoptic_evaluator.synchronize_between_processes() + + # accumulate predictions from all images + if coco_evaluator is not None: + coco_evaluator.accumulate() + coco_evaluator.summarize() + panoptic_res = None + if panoptic_evaluator is not None: + panoptic_res = panoptic_evaluator.summarize() + stats = {k: meter.global_avg for k, meter in metric_logger.meters.items()} + if coco_evaluator is not None: + if "bbox" in postprocessors.keys(): + stats["coco_eval_bbox"] = coco_evaluator.coco_eval["bbox"].stats.tolist() + if "segm" in postprocessors.keys(): + stats["coco_eval_masks"] = coco_evaluator.coco_eval["segm"].stats.tolist() + if panoptic_res is not None: + stats["PQ_all"] = panoptic_res["All"] + stats["PQ_th"] = panoptic_res["Things"] + stats["PQ_st"] = panoptic_res["Stuff"] + return stats, coco_evaluator + diff --git a/perception_models/apps/detection/DETA_pe/engine_tta.py b/perception_models/apps/detection/DETA_pe/engine_tta.py new file mode 100644 index 0000000000000000000000000000000000000000..3f95f707aaac0ae4bfa8b598b3c2dbb4c41fa6e1 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/engine_tta.py @@ -0,0 +1,239 @@ +# ------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------ +# Modified from DETR (https://github.com/facebookresearch/detr) +# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved +# ------------------------------------------------------------------------ + +""" +Train and eval functions used in main.py +""" +import math +import os +import sys +from typing import Iterable + +import torch +import util.misc as utils +from datasets.coco_eval import CocoEvaluator, convert_to_xywh +from datasets.data_prefetcher import data_prefetcher +from datasets.panoptic_eval import PanopticEvaluator +from models.utils_softnms import batched_soft_nms +from util.misc import NestedTensor + + +# Make sure this is consistent with datasets/coco.py +# TODO: make it configurable +SCALE_RANGES_DICT = { + 1728: [[0, 10000], [32, 10000], [32, 10000],], + 1824: [[0, 10000], [0, 10000], [64, 10000], [64, 10000],], +} + + +def filter_boxes(boxes, min_scale, max_scale): + """ + boxes: (N, 4) shape + """ + w = boxes[:, 2] - boxes[:, 0] + h = boxes[:, 3] - boxes[:, 1] + keep = (w * h > min_scale * min_scale) & (w * h < max_scale * max_scale) + return keep + + +@torch.no_grad() +def evaluate_tta( + model_no_ema, + criterion, + postprocessors, + data_loader, + base_ds, + device, + output_dir, + test_hflip_aug, + tta, + soft_nms, + ema=None, + save_result=False, + save_result_dir="", + soft_nms_method="quad", + nms_thresh=0.7, + quad_scale=0.5, + lsj_img_size=1824, +): + model = model_no_ema if ema is None else ema + model.eval() + criterion.eval() + + metric_logger = utils.MetricLogger(delimiter=" ") + metric_logger.add_meter( + "class_error", utils.SmoothedValue(window_size=1, fmt="{value:.2f}") + ) + header = "Test:" + + iou_types = tuple(k for k in ("segm", "bbox") if k in postprocessors.keys()) + coco_evaluator = CocoEvaluator(base_ds, iou_types) + # coco_evaluator.coco_eval[iou_types[0]].params.iouThrs = [0, 0.1, 0.5, 0.75] + + SCALE_RANGES = SCALE_RANGES_DICT[lsj_img_size] + IMAGE_SIZE = [lsj_img_size for _ in range(len(SCALE_RANGES))] + + prediction_list = [] + for samples, targets in metric_logger.log_every(data_loader, 10, header): + samples = samples.to(device) + targets = [{k: v.to(device) for k, v in t.items()} for t in targets] + + orig_target_sizes = torch.stack([t["orig_size"] for t in targets], dim=0) + metric_logger.update(loss=0, class_error=0, loss_bbox=0, loss_ce=0) + ########################### Begin of inference_one_image ########################### + if tta: + assert samples.tensors.shape[0] == 1, "tta only supports batch size 1" + assert ( + samples.tensors.shape[1] % 3 == 0 + ), "tta requires dimensions of samples.tensors to be divisible by 3" + + all_boxes = [] + all_scores = [] + all_classes = [] + + num_scales = samples.tensors.shape[1] // 3 + for scale_ind in range(num_scales): + first_samples = NestedTensor( + samples.tensors[ + :, + scale_ind * 3 : (scale_ind + 1) * 3, + : IMAGE_SIZE[scale_ind // 2], + : IMAGE_SIZE[scale_ind // 2], + ], + samples.mask[ + :, + scale_ind, + : IMAGE_SIZE[scale_ind // 2], + : IMAGE_SIZE[scale_ind // 2], + ], + ) + + if scale_ind % 2 == 0: + ######## no flip ####### + outputs = model(first_samples) + noaug_results = postprocessors["bbox"]( + outputs, + orig_target_sizes, + soft_nms=soft_nms, + method=soft_nms_method, + nms_thresh=nms_thresh, + quad_scale=quad_scale, + ) + keep = filter_boxes( + noaug_results[0]["boxes"], *SCALE_RANGES[scale_ind // 2] + ) + all_boxes.append(noaug_results[0]["boxes"][keep]) + all_scores.append(noaug_results[0]["scores"][keep]) + all_classes.append(noaug_results[0]["labels"][keep]) + else: + ######## flipped ####### + flipped_outputs = model(first_samples) + flipped_pred_logits = flipped_outputs["pred_logits"] + flipped_pred_boxes = flipped_outputs["pred_boxes"] + reflipped_pred_boxes = flipped_pred_boxes[ + :, :, [0, 1, 2, 3] + ] * torch.as_tensor([-1, 1, 1, 1]).to( + flipped_pred_boxes.device + ) + torch.as_tensor( + [1, 0, 0, 0] + ).to( + flipped_pred_boxes.device + ) + new_outputs = {} + new_outputs["pred_logits"] = flipped_pred_logits + new_outputs["pred_boxes"] = reflipped_pred_boxes + new_results = postprocessors["bbox"]( + new_outputs, + orig_target_sizes, + soft_nms=soft_nms, + method=soft_nms_method, + nms_thresh=nms_thresh, + quad_scale=quad_scale, + ) + keep = filter_boxes( + new_results[0]["boxes"], *SCALE_RANGES[scale_ind // 2] + ) + all_boxes.append(new_results[0]["boxes"][keep]) + all_scores.append(new_results[0]["scores"][keep]) + all_classes.append(new_results[0]["labels"][keep]) + + ######## merge ####### + all_boxes = torch.cat(all_boxes, dim=0) + all_scores = torch.cat(all_scores, dim=0) + all_classes = torch.cat(all_classes, dim=0) + + keep_inds, updated_scores = batched_soft_nms( + all_boxes, + all_scores, + all_classes, + method=soft_nms_method, + threshold=nms_thresh, + quad_scale=quad_scale, + ) + merged_scores = updated_scores + merged_classes = all_classes[keep_inds] + merged_boxes = all_boxes[keep_inds] + + results = [ + { + "boxes": merged_boxes, + "scores": merged_scores, + "labels": merged_classes, + } + ] + else: + outputs = model(samples) + results = postprocessors["bbox"](outputs, orig_target_sizes) + + ########################### End of inference_one_image ########################### + res = { + target["image_id"].item(): output + for target, output in zip(targets, results) + } + if coco_evaluator is not None: + coco_evaluator.update(res) + + for target, output in zip(targets, results): + res_cpu = { + target["image_id"].item(): { + "boxes": output["boxes"].cpu(), + "labels": output["labels"].cpu(), + "scores": output["scores"].cpu(), + } + } + prediction_list.append(res_cpu) + + # gather the stats from all processes + metric_logger.synchronize_between_processes() + print("Averaged stats:", metric_logger) + + if save_result: + from torch import distributed as dist + + os.makedirs(save_result_dir, exist_ok=True) + + rank = dist.get_rank() + torch.save( + prediction_list, + os.path.join(save_result_dir, f"val2017_prediction_{rank}.pth"), + ) + + if coco_evaluator is not None: + coco_evaluator.synchronize_between_processes() + + # accumulate predictions from all images + if coco_evaluator is not None: + coco_evaluator.accumulate() + coco_evaluator.summarize() + + stats = {k: meter.global_avg for k, meter in metric_logger.meters.items()} + if coco_evaluator is not None: + if "bbox" in postprocessors.keys(): + stats["coco_eval_bbox"] = coco_evaluator.coco_eval["bbox"].stats.tolist() + return stats, coco_evaluator diff --git a/perception_models/apps/detection/DETA_pe/main.py b/perception_models/apps/detection/DETA_pe/main.py new file mode 100644 index 0000000000000000000000000000000000000000..9cb79d8e292811f559daf1f26ea35f0075fa0056 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/main.py @@ -0,0 +1,754 @@ +# Modified from +# ------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------ +# Modified from DETR (https://github.com/facebookresearch/detr) +# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved +# ------------------------------------------------------------------------ + + +import argparse +import datetime +import json +import os +import random +import time +from copy import deepcopy +from pathlib import Path + +import datasets +import datasets.samplers as samplers + +import numpy as np +import torch +import util.misc as utils +from datasets import build_dataset, get_coco_api_from_dataset +from engine import evaluate, train_one_epoch +from engine_tta import evaluate_tta +from models import build_model +from torch.utils.data import DataLoader +from util.ema import requires_grad, update_ema + + +def get_args_parser(): + parser = argparse.ArgumentParser("Deformable DETR Detector", add_help=False) + parser.add_argument("--lr", default=2e-4, type=float) + parser.add_argument( + "--lr_backbone_names", default=["backbone.0"], type=str, nargs="+" + ) + parser.add_argument("--lr_backbone", default=2e-5, type=float) + parser.add_argument( + "--lr_linear_proj_names", + default=["reference_points", "sampling_offsets"], + type=str, + nargs="+", + ) + parser.add_argument("--lr_linear_proj_mult", default=0.1, type=float) + parser.add_argument("--batch_size", default=2, type=int) + parser.add_argument("--weight_decay", default=1e-4, type=float) + parser.add_argument("--epochs", default=50, type=int) + parser.add_argument("--eval_per_epochs", default=1, type=int) + parser.add_argument("--save_per_epochs", default=1, type=int) + parser.add_argument("--lr_drop", default=40, type=int) + parser.add_argument("--lr_drop_epochs", default=None, type=int, nargs="+") + parser.add_argument( + "--clip_max_norm", default=0.1, type=float, help="gradient clipping max norm" + ) + + parser.add_argument("--sgd", action="store_true") + parser.add_argument("--ema", action="store_true") + parser.add_argument("--ema_decay", default=0.999, type=float) + + # Variants of Deformable DETR + parser.add_argument("--with_box_refine", default=False, action="store_true") + parser.add_argument("--two_stage", default=False, action="store_true") + + # Model parameters + parser.add_argument( + "--frozen_weights", + type=str, + default=None, + help="Path to the pretrained model. If set, only the mask head will be trained", + ) + + # * Backbone + parser.add_argument( + "--backbone", + default="resnet50", + type=str, + help="Name of the convolutional backbone to use", + ) + parser.add_argument( + "--backbone_size", + default="Gwin384", + type=str, + help="backbone size", + ) + parser.add_argument( + "--backbone_path", + default="", + type=str, + ) + parser.add_argument( + "--backbone_lrd", + default=1.0, + type=float, + ) + parser.add_argument( + "--backbone_layers", + default=12, + type=int, + ) + parser.add_argument( + "--backbone_init_values", + default=0.0, + type=float, + ) + parser.add_argument( + "--backbone_tile_posemb", + default=False, + type=bool, + ) + parser.add_argument( + "--backbone_use_act_checkpoint", + action="store_true", + help="If true, we use act_checkpoint in backbone", + ) + parser.add_argument( + "--backbone_act_checkpoint_ratio", + default=1.0, + type=float, + ) + parser.add_argument( + "--backbone_tta_rope", + action="store_true", + ) + parser.add_argument( + "--backbone_multi_layer", + action="store_true", + ) + + parser.add_argument( + "--backbone_win_aug", + action="store_true", + ) + + parser.add_argument( + "--backbone_dp", + default=-1.0, + type=float, + ) + + parser.add_argument( + "--bf16", + action="store_true", + ) + parser.add_argument( + "--fp16", + action="store_true", + ) + parser.add_argument( + "--dilation", + action="store_true", + help="If true, we replace stride with dilation in the last convolutional block (DC5)", + ) + parser.add_argument( + "--position_embedding", + default="sine", + type=str, + choices=("sine", "learned"), + help="Type of positional embedding to use on top of the image features", + ) + parser.add_argument( + "--position_embedding_scale", + default=2 * np.pi, + type=float, + help="position / size * scale", + ) + parser.add_argument( + "--num_feature_levels", default=4, type=int, help="number of feature levels" + ) + + # * Transformer + parser.add_argument( + "--enc_layers", + default=6, + type=int, + help="Number of encoding layers in the transformer", + ) + parser.add_argument( + "--dec_layers", + default=6, + type=int, + help="Number of decoding layers in the transformer", + ) + parser.add_argument( + "--dim_feedforward", + default=1024, + type=int, + help="Intermediate size of the feedforward layers in the transformer blocks", + ) + parser.add_argument( + "--hidden_dim", + default=256, + type=int, + help="Size of the embeddings (dimension of the transformer)", + ) + parser.add_argument( + "--dropout", default=0.1, type=float, help="Dropout applied in the transformer" + ) + parser.add_argument( + "--nheads", + default=8, + type=int, + help="Number of attention heads inside the transformer's attentions", + ) + parser.add_argument( + "--num_queries", default=300, type=int, help="Number of query slots" + ) + parser.add_argument("--dec_n_points", default=4, type=int) + parser.add_argument("--enc_n_points", default=4, type=int) + + # * Segmentation + parser.add_argument( + "--masks", + action="store_true", + help="Train segmentation head if the flag is provided", + ) + + # Loss + parser.add_argument( + "--no_aux_loss", + dest="aux_loss", + action="store_false", + help="Disables auxiliary decoding losses (loss at each layer)", + ) + parser.add_argument("--use_fed_loss", action="store_true") + + # * Matcher + parser.add_argument("--assign_first_stage", action="store_true") + parser.add_argument("--assign_second_stage", action="store_true") + parser.add_argument( + "--set_cost_class", + default=2, + type=float, + help="Class coefficient in the matching cost", + ) + parser.add_argument( + "--set_cost_bbox", + default=5, + type=float, + help="L1 box coefficient in the matching cost", + ) + parser.add_argument( + "--set_cost_giou", + default=2, + type=float, + help="giou box coefficient in the matching cost", + ) + + # * Loss coefficients + parser.add_argument("--mask_loss_coef", default=1, type=float) + parser.add_argument("--dice_loss_coef", default=1, type=float) + parser.add_argument("--cls_loss_coef", default=2, type=float) + parser.add_argument("--bbox_loss_coef", default=5, type=float) + parser.add_argument("--giou_loss_coef", default=2, type=float) + parser.add_argument("--focal_alpha", default=0.25, type=float) + + # dataset parameters + parser.add_argument("--new_mean_std", action="store_true") + parser.add_argument("--dataset_file", default="coco") + parser.add_argument("--coco_path", default="./data/coco", type=str) + parser.add_argument("--coco_panoptic_path", type=str) + parser.add_argument("--remove_difficult", action="store_true") + parser.add_argument("--bigger", action="store_true") + parser.add_argument("--lsj", action="store_true") + parser.add_argument("--lsj_ms", action="store_true") + + parser.add_argument("--lsj_img_size", default=1024, type=int) + parser.add_argument("--lsj_img_train_min", default=480, type=int) + parser.add_argument("--lsj_img_size_max", default=-1, type=int) + parser.add_argument("--lsj_strong_aug", action="store_true") + + parser.add_argument("--save_result", action="store_true") + parser.add_argument("--save_result_dir", default="", type=str) + parser.add_argument("--test_hflip_aug", action="store_true") + parser.add_argument("--tta", action="store_true") + parser.add_argument("--soft_nms", action="store_true") + parser.add_argument("--soft_nms_method", default="quad", type=str) + parser.add_argument("--nms_thresh", default=0.7, type=float) + parser.add_argument("--quad_scale", default=0.5, type=float) + parser.add_argument( + "--output_dir", default="", help="path where to save, empty for no saving" + ) + parser.add_argument( + "--device", default="cuda", help="device to use for training / testing" + ) + parser.add_argument("--seed", default=42, type=int) + parser.add_argument("--resume", default="", help="resume from checkpoint") + parser.add_argument("--auto_resume", action="store_true") + + parser.add_argument( + "--resume_norope", + action="store_true", + help="resume from checkpoint without rope params", + ) + parser.add_argument("--finetune", default="", help="finetune from checkpoint") + parser.add_argument("--keep_class_embed", action="store_true") + parser.add_argument( + "--start_epoch", default=0, type=int, metavar="N", help="start epoch" + ) + parser.add_argument("--eval", action="store_true") + parser.add_argument("--num_workers", default=8, type=int) + parser.add_argument( + "--cache_mode", + default=False, + action="store_true", + help="whether to cache images on memory", + ) + + return parser + + +# lr_backbone_names = ["backbone.0", "backbone.neck", "input_proj", "transformer.encoder"] +def match_name_keywords(n, name_keywords): + out = False + for b in name_keywords: + if b in n: + out = True + break + return out + + +def get_vit_lr_decay_rate_vev01(name, lr_decay_rate=1.0, num_layers=12): + layer_id = num_layers + 1 + if ".positional_embedding" in name or ".conv1" in name or ".ln_pre" in name: + layer_id = 0 + elif ".resblocks." in name: + layer_id = int(name[name.find(".resblocks.") :].split(".")[2]) + 1 + return lr_decay_rate ** (num_layers + 1 - layer_id) + + +def custom_lr(model_without_ddp, args): + param_dicts = [ + { + "params": [ + p + for n, p in model_without_ddp.named_parameters() + if not match_name_keywords(n, args.lr_backbone_names) + and not match_name_keywords(n, args.lr_linear_proj_names) + and p.requires_grad + ], + "lr": args.lr, + }, + { + "params": [ + p + for n, p in model_without_ddp.named_parameters() + if match_name_keywords(n, args.lr_linear_proj_names) and p.requires_grad + ], + "lr": args.lr * args.lr_linear_proj_mult, + }, + ] + if "vev01" in args.backbone: + for p_key, p_value in model_without_ddp.named_parameters(): + if ( + match_name_keywords(p_key, args.lr_backbone_names) + and p_value.requires_grad + ): + p_lr = args.lr_backbone * get_vit_lr_decay_rate_vev01( + p_key, args.backbone_lrd, args.backbone_layers + ) + param_dicts.append( + { + "params": [p_value], + "lr": p_lr, + } + ) + print(f"param_name: {p_key}, lr: {p_lr}") + else: + param_groups_backbone = { + "params": [ + p + for n, p in model_without_ddp.named_parameters() + if match_name_keywords(n, args.lr_backbone_names) and p.requires_grad + ], + "lr": args.lr_backbone, + } + param_dicts.append(param_groups_backbone) + + return param_dicts + + +def main(args): + utils.init_distributed_mode(args) + print("git:\n {}\n".format(utils.get_sha())) + + if args.frozen_weights is not None: + assert args.masks, "Frozen training is meant for segmentation only" + print(args) + + device = torch.device(args.device) + + # fix the seed for reproducibility + seed = args.seed + utils.get_rank() + torch.manual_seed(seed) + np.random.seed(seed) + random.seed(seed) + + model, criterion, postprocessors = build_model(args) + model.to(device) + + model_without_ddp = model + n_parameters = sum(p.numel() for p in model.parameters() if p.requires_grad) + print("model:", model_without_ddp) + for n, p in model_without_ddp.named_parameters(): + print(n) + print("number of params:", n_parameters) + + if args.ema: + ema = deepcopy(model).to(device) + requires_grad(ema, False) + print(f"EMA Parameters: {sum(p.numel() for p in ema.parameters()):,}") + + dataset_train = build_dataset(image_set="train", args=args) + dataset_val = build_dataset(image_set="val", args=args) + + if args.distributed: + if args.cache_mode: + sampler_train = samplers.NodeDistributedSampler(dataset_train) + sampler_val = samplers.NodeDistributedSampler(dataset_val, shuffle=False) + else: + if args.dataset_file == "lvis": + sampler_train = samplers.RepeatFactorTrainingSampler(dataset_train) + else: + sampler_train = samplers.DistributedSampler(dataset_train) + sampler_val = samplers.DistributedSampler(dataset_val, shuffle=False) + else: + sampler_train = torch.utils.data.RandomSampler(dataset_train) + sampler_val = torch.utils.data.SequentialSampler(dataset_val) + + batch_sampler_train = torch.utils.data.BatchSampler( + sampler_train, args.batch_size, drop_last=True + ) + if args.lsj_ms: + collator = utils.CollatorLSJMultiscale(args.lsj_img_size, args.tta) + elif args.lsj: + lsj_img_size_colla = ( + args.lsj_img_size_max if args.lsj_img_size_max > 0 else args.lsj_img_size + ) + collator = utils.CollatorLSJ(lsj_img_size_colla, args.tta) + else: + collator = utils.collate_fn + + data_loader_train = DataLoader( + dataset_train, + batch_sampler=batch_sampler_train, + collate_fn=collator, + num_workers=args.num_workers, + pin_memory=True, + ) + data_loader_val = DataLoader( + dataset_val, + args.batch_size, + sampler=sampler_val, + drop_last=False, + collate_fn=collator, + num_workers=args.num_workers, + pin_memory=True, + ) + + param_dicts = custom_lr(model_without_ddp, args) + + if args.sgd: + optimizer = torch.optim.SGD( + param_dicts, lr=args.lr, momentum=0.9, weight_decay=args.weight_decay + ) + else: + optimizer = torch.optim.AdamW( + param_dicts, lr=args.lr, weight_decay=args.weight_decay + ) + lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, args.lr_drop) + + if args.distributed: + model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu]) + model_without_ddp = model.module + + if args.dataset_file == "coco_panoptic": + # We also evaluate AP during panoptic training, on original coco DS + coco_val = datasets.coco.build("val", args) + base_ds = get_coco_api_from_dataset(coco_val) + else: + base_ds = get_coco_api_from_dataset(dataset_val) + + if args.frozen_weights is not None: + checkpoint = torch.load(args.frozen_weights, map_location="cpu") + model_without_ddp.detr.load_state_dict(checkpoint["model"]) + + if args.tta: + evaluate_fn = evaluate_tta + else: + evaluate_fn = evaluate + + output_dir = Path(args.output_dir) + if args.auto_resume: + resumed_ckpt = os.path.join(args.output_dir, "checkpoint.pth") + if os.path.exists(resumed_ckpt): + args.resume = resumed_ckpt + args.finetune = None + + if args.finetune: + checkpoint = torch.load(args.finetune, map_location="cpu") + state_dict = checkpoint["model"] + for k in list(state_dict.keys()): + if "class_embed" in k and not args.keep_class_embed: + print("removing", k) + del state_dict[k] + if "freqs" in k: + print("removing", k) + del state_dict[k] + + missing_keys, unexpected_keys = model_without_ddp.load_state_dict( + state_dict, strict=False + ) + unexpected_keys = [ + k + for k in unexpected_keys + if not (k.endswith("total_params") or k.endswith("total_ops")) + ] + if len(missing_keys) > 0: + print("Missing Keys: {}".format(missing_keys)) + if len(unexpected_keys) > 0: + print("Unexpected Keys: {}".format(unexpected_keys)) + + if "epoch" in checkpoint: + print("finetuning from epoch", checkpoint["epoch"]) + + if args.ema: + ema.load_state_dict( + checkpoint["ema"] if "ema" in checkpoint else state_dict, strict=False + ) + + if args.resume: + print("Resuming training from {}".format(args.resume)) + if args.resume.startswith("https"): + checkpoint = torch.hub.load_state_dict_from_url( + args.resume, map_location="cpu", check_hash=True + ) + else: + checkpoint = torch.load(args.resume, map_location="cpu") + + if args.resume_norope: + state_dict = checkpoint["model"] + for k in list(state_dict.keys()): + if "freqs" in k: + print("removing", k) + del state_dict[k] + + missing_keys, unexpected_keys = model_without_ddp.load_state_dict( + state_dict, strict=False + ) + if args.ema: + ema.load_state_dict( + checkpoint["ema"] if "ema" in checkpoint else state_dict, + strict=False, + ) + else: + missing_keys, unexpected_keys = model_without_ddp.load_state_dict( + checkpoint["model"], strict=False + ) + if args.ema: + ema.load_state_dict( + checkpoint["ema"] if "ema" in checkpoint else state_dict, + strict=False, + ) + unexpected_keys = [ + k + for k in unexpected_keys + if not (k.endswith("total_params") or k.endswith("total_ops")) + ] + if len(missing_keys) > 0: + print("Missing Keys: {}".format(missing_keys)) + if len(unexpected_keys) > 0: + print("Unexpected Keys: {}".format(unexpected_keys)) + if ( + not args.eval + and "optimizer" in checkpoint + and "lr_scheduler" in checkpoint + and "epoch" in checkpoint + ): + import copy + + p_groups = copy.deepcopy(optimizer.param_groups) + optimizer.load_state_dict(checkpoint["optimizer"]) + for pg, pg_old in zip(optimizer.param_groups, p_groups): + pg["lr"] = pg_old["lr"] + pg["initial_lr"] = pg_old["initial_lr"] + print(optimizer.param_groups) + lr_scheduler.load_state_dict(checkpoint["lr_scheduler"]) + # todo: this is a hack for doing experiment that resume from checkpoint and also modify lr scheduler (e.g., decrease lr in advance). + args.override_resumed_lr_drop = True + if args.override_resumed_lr_drop: + print( + "Warning: (hack) args.override_resumed_lr_drop is set to True, so args.lr_drop would override lr_drop in resumed lr_scheduler." + ) + lr_scheduler.step_size = args.lr_drop + lr_scheduler.base_lrs = list( + map(lambda group: group["initial_lr"], optimizer.param_groups) + ) + lr_scheduler.step(lr_scheduler.last_epoch) + args.start_epoch = checkpoint["epoch"] + 1 + # check the resumed model + if not args.eval: + test_stats, coco_evaluator = evaluate_fn( + model, + criterion, + postprocessors, + data_loader_val, + base_ds, + device, + args.output_dir, + args.test_hflip_aug, + args.tta, + args.soft_nms, + ema if args.ema else None, + args.save_result, + args.save_result_dir, + soft_nms_method=args.soft_nms_method, + nms_thresh=args.nms_thresh, + quad_scale=args.quad_scale, + lsj_img_size=args.lsj_img_size, + ) + torch.cuda.empty_cache() + + if args.eval: + test_stats, coco_evaluator = evaluate_fn( + model, + criterion, + postprocessors, + data_loader_val, + base_ds, + device, + args.output_dir, + args.test_hflip_aug, + args.tta, + args.soft_nms, + ema if args.ema else None, + args.save_result, + args.save_result_dir, + soft_nms_method=args.soft_nms_method, + nms_thresh=args.nms_thresh, + quad_scale=args.quad_scale, + lsj_img_size=args.lsj_img_size, + ) + + if args.output_dir: + utils.save_on_master( + coco_evaluator.coco_eval["bbox"].eval, output_dir / "eval.pth" + ) + return + + print("Start training") + start_time = time.time() + if args.ema: + ema.eval() # EMA model should always be in eval mode + for epoch in range(args.start_epoch, args.epochs): + if args.distributed: + sampler_train.set_epoch(epoch) + train_stats = train_one_epoch( + model, + criterion, + data_loader_train, + optimizer, + device, + epoch, + args.clip_max_norm, + ema if args.ema else None, + ema_decay=args.ema_decay, + ) + lr_scheduler.step() + if args.output_dir: + checkpoint_paths = [output_dir / "checkpoint.pth"] + # extra checkpoint before LR drop and every 5 epochs + if ( + (epoch + 1) % args.lr_drop == 0 + or (epoch + 1) % args.save_per_epochs == 0 + or epoch + 1 == args.epochs + ): + checkpoint_paths.append(output_dir / f"checkpoint{epoch:04}.pth") + for checkpoint_path in checkpoint_paths: + ckpt_dict = { + "model": model_without_ddp.state_dict(), + "optimizer": optimizer.state_dict(), + "lr_scheduler": lr_scheduler.state_dict(), + "epoch": epoch, + "args": args, + } + if args.ema: + ckpt_dict["ema"] = ema.state_dict() + utils.save_on_master( + ckpt_dict, + checkpoint_path, + ) + + torch.cuda.empty_cache() + if epoch % args.eval_per_epochs == 0 or epoch + 1 == args.epochs: + test_stats, coco_evaluator = evaluate_fn( + model, + criterion, + postprocessors, + data_loader_val, + base_ds, + device, + args.output_dir, + args.test_hflip_aug, + args.tta, + args.soft_nms, + ema if args.ema else None, + args.save_result, + args.save_result_dir, + soft_nms_method=args.soft_nms_method, + nms_thresh=args.nms_thresh, + quad_scale=args.quad_scale, + lsj_img_size=args.lsj_img_size, + ) + log_stats = { + **{f"train_{k}": v for k, v in train_stats.items()}, + **{f"test_{k}": v for k, v in test_stats.items()}, + "epoch": epoch, + "n_parameters": n_parameters, + } + + if args.output_dir and utils.is_main_process(): + with (output_dir / "log.txt").open("a") as f: + f.write(json.dumps(log_stats) + "\n") + + # for evaluation logs + if coco_evaluator is not None: + (output_dir / "eval").mkdir(exist_ok=True) + if "bbox" in coco_evaluator.coco_eval: + filenames = ["latest.pth"] + if epoch % 50 == 0: + filenames.append(f"{epoch:03}.pth") + for name in filenames: + torch.save( + coco_evaluator.coco_eval["bbox"].eval, + output_dir / "eval" / name, + ) + torch.cuda.empty_cache() + + total_time = time.time() - start_time + total_time_str = str(datetime.timedelta(seconds=int(total_time))) + print("Training time {}".format(total_time_str)) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser( + "Deformable DETR training and evaluation script", parents=[get_args_parser()] + ) + args = parser.parse_args() + if args.output_dir: + Path(args.output_dir).mkdir(parents=True, exist_ok=True) + main(args) diff --git a/perception_models/apps/detection/DETA_pe/models/__init__.py b/perception_models/apps/detection/DETA_pe/models/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..9a59c33484884af523013d1ed2ef57032646336a --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/models/__init__.py @@ -0,0 +1,15 @@ +# ------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------ +# Modified from DETR (https://github.com/facebookresearch/detr) +# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved +# ------------------------------------------------------------------------ + +from .deformable_detr import build + + +def build_model(args): + return build(args) + diff --git a/perception_models/apps/detection/DETA_pe/models/assigner.py b/perception_models/apps/detection/DETA_pe/models/assigner.py new file mode 100644 index 0000000000000000000000000000000000000000..0238c8aff489fa6a36900faad9939f3d5f57a042 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/models/assigner.py @@ -0,0 +1,378 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +# Modified by Jeffrey Ouyang-Zhang + +from typing import List + +import torch +import torch.nn as nn + +from util.box_ops import ( + box_cxcywh_to_xyxy, + box_iou, + box_xyxy_to_cxcywh, + generalized_box_iou, +) + + +# from https://github.com/facebookresearch/detectron2/blob/cbbc1ce26473cb2a5cc8f58e8ada9ae14cb41052/detectron2/layers/wrappers.py#L100 +def nonzero_tuple(x): + """ + A 'as_tuple=True' version of torch.nonzero to support torchscript. + because of https://github.com/pytorch/pytorch/issues/38718 + """ + if torch.jit.is_scripting(): + if x.dim() == 0: + return x.unsqueeze(0).nonzero().unbind(1) + return x.nonzero().unbind(1) + else: + return x.nonzero(as_tuple=True) + + +# from https://github.com/facebookresearch/detectron2/blob/9921a2caa585d4fa66c4b534b6fab6e74d89b582/detectron2/modeling/matcher.py#L9 +class Matcher(object): + """ + This class assigns to each predicted "element" (e.g., a box) a ground-truth + element. Each predicted element will have exactly zero or one matches; each + ground-truth element may be matched to zero or more predicted elements. + + The matching is determined by the MxN match_quality_matrix, that characterizes + how well each (ground-truth, prediction)-pair match each other. For example, + if the elements are boxes, this matrix may contain box intersection-over-union + overlap values. + + The matcher returns (a) a vector of length N containing the index of the + ground-truth element m in [0, M) that matches to prediction n in [0, N). + (b) a vector of length N containing the labels for each prediction. + """ + + def __init__( + self, + thresholds: List[float], + labels: List[int], + allow_low_quality_matches: bool = False, + ): + """ + Args: + thresholds (list): a list of thresholds used to stratify predictions + into levels. + labels (list): a list of values to label predictions belonging at + each level. A label can be one of {-1, 0, 1} signifying + {ignore, negative class, positive class}, respectively. + allow_low_quality_matches (bool): if True, produce additional matches + for predictions with maximum match quality lower than high_threshold. + See set_low_quality_matches_ for more details. + + For example, + thresholds = [0.3, 0.5] + labels = [0, -1, 1] + All predictions with iou < 0.3 will be marked with 0 and + thus will be considered as false positives while training. + All predictions with 0.3 <= iou < 0.5 will be marked with -1 and + thus will be ignored. + All predictions with 0.5 <= iou will be marked with 1 and + thus will be considered as true positives. + """ + # Add -inf and +inf to first and last position in thresholds + thresholds = thresholds[:] + assert thresholds[0] > 0 + thresholds.insert(0, -float("inf")) + thresholds.append(float("inf")) + # Currently torchscript does not support all + generator + assert all( + [low <= high for (low, high) in zip(thresholds[:-1], thresholds[1:])] + ), thresholds + assert all([l in [-1, 0, 1] for l in labels]) + assert len(labels) == len(thresholds) - 1 + self.thresholds = thresholds + self.labels = labels + self.allow_low_quality_matches = allow_low_quality_matches + + def __call__(self, match_quality_matrix): + """ + Args: + match_quality_matrix (Tensor[float]): an MxN tensor, containing the + pairwise quality between M ground-truth elements and N predicted + elements. All elements must be >= 0 (due to the us of `torch.nonzero` + for selecting indices in :meth:`set_low_quality_matches_`). + + Returns: + matches (Tensor[int64]): a vector of length N, where matches[i] is a matched + ground-truth index in [0, M) + match_labels (Tensor[int8]): a vector of length N, where pred_labels[i] indicates + whether a prediction is a true or false positive or ignored + """ + assert match_quality_matrix.dim() == 2 + if match_quality_matrix.numel() == 0: + default_matches = match_quality_matrix.new_full( + (match_quality_matrix.size(1),), 0, dtype=torch.int64 + ) + # When no gt boxes exist, we define IOU = 0 and therefore set labels + # to `self.labels[0]`, which usually defaults to background class 0 + # To choose to ignore instead, can make labels=[-1,0,-1,1] + set appropriate thresholds + default_match_labels = match_quality_matrix.new_full( + (match_quality_matrix.size(1),), self.labels[0], dtype=torch.int8 + ) + return default_matches, default_match_labels + + assert torch.all(match_quality_matrix >= 0) + + # match_quality_matrix is M (gt) x N (predicted) + # Max over gt elements (dim 0) to find best gt candidate for each prediction + matched_vals, matches = match_quality_matrix.max(dim=0) + + match_labels = matches.new_full(matches.size(), 1, dtype=torch.int8) + + for l, low, high in zip(self.labels, self.thresholds[:-1], self.thresholds[1:]): + low_high = (matched_vals >= low) & (matched_vals < high) + match_labels[low_high] = l + + if self.allow_low_quality_matches: + self.set_low_quality_matches_(match_labels, match_quality_matrix) + + return matches, match_labels + + def set_low_quality_matches_(self, match_labels, match_quality_matrix): + """ + Produce additional matches for predictions that have only low-quality matches. + Specifically, for each ground-truth G find the set of predictions that have + maximum overlap with it (including ties); for each prediction in that set, if + it is unmatched, then match it to the ground-truth G. + + This function implements the RPN assignment case (i) in Sec. 3.1.2 of + :paper:`Faster R-CNN`. + """ + # For each gt, find the prediction with which it has highest quality + highest_quality_foreach_gt, _ = match_quality_matrix.max(dim=1) + # Find the highest quality match available, even if it is low, including ties. + # Note that the matches qualities must be positive due to the use of + # `torch.nonzero`. + _, pred_inds_with_highest_quality = nonzero_tuple( + match_quality_matrix == highest_quality_foreach_gt[:, None] + ) + # If an anchor was labeled positive only due to a low-quality match + # with gt_A, but it has larger overlap with gt_B, it's matched index will still be gt_B. + # This follows the implementation in Detectron, and is found to have no significant impact. + match_labels[pred_inds_with_highest_quality] = 1 + + +# from https://github.com/facebookresearch/detectron2/blob/cbbc1ce26473cb2a5cc8f58e8ada9ae14cb41052/detectron2/modeling/sampling.py#L9 +def subsample_labels( + labels: torch.Tensor, num_samples: int, positive_fraction: float, bg_label: int +): + """ + Return `num_samples` (or fewer, if not enough found) + random samples from `labels` which is a mixture of positives & negatives. + It will try to return as many positives as possible without + exceeding `positive_fraction * num_samples`, and then try to + fill the remaining slots with negatives. + + Args: + labels (Tensor): (N, ) label vector with values: + * -1: ignore + * bg_label: background ("negative") class + * otherwise: one or more foreground ("positive") classes + num_samples (int): The total number of labels with value >= 0 to return. + Values that are not sampled will be filled with -1 (ignore). + positive_fraction (float): The number of subsampled labels with values > 0 + is `min(num_positives, int(positive_fraction * num_samples))`. The number + of negatives sampled is `min(num_negatives, num_samples - num_positives_sampled)`. + In order words, if there are not enough positives, the sample is filled with + negatives. If there are also not enough negatives, then as many elements are + sampled as is possible. + bg_label (int): label index of background ("negative") class. + + Returns: + pos_idx, neg_idx (Tensor): + 1D vector of indices. The total length of both is `num_samples` or fewer. + """ + positive = nonzero_tuple((labels != -1) & (labels != bg_label))[0] + negative = nonzero_tuple(labels == bg_label)[0] + + num_pos = int(num_samples * positive_fraction) + # protect against not enough positive examples + num_pos = min(positive.numel(), num_pos) + num_neg = num_samples - num_pos + # protect against not enough negative examples + num_neg = min(negative.numel(), num_neg) + + # randomly select positive and negative examples + perm1 = torch.randperm(positive.numel(), device=positive.device)[:num_pos] + perm2 = torch.randperm(negative.numel(), device=negative.device)[:num_neg] + + pos_idx = positive[perm1] + neg_idx = negative[perm2] + return pos_idx, neg_idx + + +def sample_topk_per_gt(pr_inds, gt_inds, iou, k): + if len(gt_inds) == 0: + return pr_inds, gt_inds + # find topk matches for each gt + gt_inds2, counts = gt_inds.unique(return_counts=True) + scores, pr_inds2 = iou[gt_inds2].topk(k, dim=1) + gt_inds2 = gt_inds2[:, None].repeat(1, k) + + # filter to as many matches that gt has + pr_inds3 = torch.cat([pr[:c] for c, pr in zip(counts, pr_inds2)]) + gt_inds3 = torch.cat([gt[:c] for c, gt in zip(counts, gt_inds2)]) + return pr_inds3, gt_inds3 + + +# modified from https://github.com/facebookresearch/detectron2/blob/cbbc1ce26473cb2a5cc8f58e8ada9ae14cb41052/detectron2/modeling/roi_heads/roi_heads.py#L123 +class Stage2Assigner(nn.Module): + def __init__(self, num_queries, max_k=4): + super().__init__() + self.positive_fraction = 0.25 + self.bg_label = 400 # number > 91 to filter out later + self.batch_size_per_image = num_queries + self.proposal_matcher = Matcher( + thresholds=[0.6], labels=[0, 1], allow_low_quality_matches=True + ) + self.k = max_k + + def _sample_proposals( + self, + matched_idxs: torch.Tensor, + matched_labels: torch.Tensor, + gt_classes: torch.Tensor, + ): + """ + Based on the matching between N proposals and M groundtruth, + sample the proposals and set their classification labels. + + Args: + matched_idxs (Tensor): a vector of length N, each is the best-matched + gt index in [0, M) for each proposal. + matched_labels (Tensor): a vector of length N, the matcher's label + (one of cfg.MODEL.ROI_HEADS.IOU_LABELS) for each proposal. + gt_classes (Tensor): a vector of length M. + + Returns: + Tensor: a vector of indices of sampled proposals. Each is in [0, N). + Tensor: a vector of the same length, the classification label for + each sampled proposal. Each sample is labeled as either a category in + [0, num_classes) or the background (num_classes). + """ + has_gt = gt_classes.numel() > 0 + # Get the corresponding GT for each proposal + if has_gt: + gt_classes = gt_classes[matched_idxs] + # Label unmatched proposals (0 label from matcher) as background (label=num_classes) + gt_classes[matched_labels == 0] = self.bg_label + # Label ignore proposals (-1 label) + gt_classes[matched_labels == -1] = -1 + else: + gt_classes = torch.zeros_like(matched_idxs) + self.bg_label + + sampled_fg_idxs, sampled_bg_idxs = subsample_labels( + gt_classes, self.batch_size_per_image, self.positive_fraction, self.bg_label + ) + + sampled_idxs = torch.cat([sampled_fg_idxs, sampled_bg_idxs], dim=0) + return sampled_idxs, gt_classes[sampled_idxs] + + def forward(self, outputs, targets, return_cost_matrix=False): + # COCO categories are from 1 to 90. They set num_classes=91 and apply sigmoid. + + bs = len(targets) + indices = [] + ious = [] + for b in range(bs): + iou, _ = box_iou( + box_cxcywh_to_xyxy(targets[b]["boxes"]), + box_cxcywh_to_xyxy(outputs["init_reference"][b].detach()), + ) + matched_idxs, matched_labels = self.proposal_matcher( + iou + ) # proposal_id -> highest_iou_gt_id, proposal_id -> [1 if iou > 0.6, 0 ow] + sampled_idxs, sampled_gt_classes = ( + self._sample_proposals( # list of sampled proposal_ids, sampled_id -> [0, num_classes)+[bg_label] + matched_idxs, matched_labels, targets[b]["labels"] + ) + ) + pos_pr_inds = sampled_idxs[sampled_gt_classes != self.bg_label] + pos_gt_inds = matched_idxs[pos_pr_inds] + pos_pr_inds, pos_gt_inds = self.postprocess_indices( + pos_pr_inds, pos_gt_inds, iou + ) + indices.append((pos_pr_inds, pos_gt_inds)) + ious.append(iou) + if return_cost_matrix: + return indices, ious + return indices + + def postprocess_indices(self, pr_inds, gt_inds, iou): + return sample_topk_per_gt(pr_inds, gt_inds, iou, self.k) + + +# modified from https://github.com/facebookresearch/detectron2/blob/cbbc1ce26473cb2a5cc8f58e8ada9ae14cb41052/detectron2/modeling/proposal_generator/rpn.py#L181 +class Stage1Assigner(nn.Module): + def __init__(self, t_low=0.3, t_high=0.7, max_k=4): + super().__init__() + self.positive_fraction = 0.5 + self.batch_size_per_image = 256 + self.k = max_k + self.t_low = t_low + self.t_high = t_high + self.anchor_matcher = Matcher( + thresholds=[t_low, t_high], + labels=[0, -1, 1], + allow_low_quality_matches=True, + ) + + def _subsample_labels(self, label): + """ + Randomly sample a subset of positive and negative examples, and overwrite + the label vector to the ignore value (-1) for all elements that are not + included in the sample. + + Args: + labels (Tensor): a vector of -1, 0, 1. Will be modified in-place and returned. + """ + pos_idx, neg_idx = subsample_labels( + label, self.batch_size_per_image, self.positive_fraction, 0 + ) + # Fill with the ignore label (-1), then set positive and negative labels + label.fill_(-1) + label.scatter_(0, pos_idx, 1) + label.scatter_(0, neg_idx, 0) + return label + + def forward(self, outputs, targets): + bs = len(targets) + indices = [] + for b in range(bs): + anchors = outputs["anchors"][b] + if len(targets[b]["boxes"]) == 0: + indices.append( + ( + torch.tensor([], dtype=torch.long, device=anchors.device), + torch.tensor([], dtype=torch.long, device=anchors.device), + ) + ) + continue + iou, _ = box_iou( + box_cxcywh_to_xyxy(targets[b]["boxes"]), + box_cxcywh_to_xyxy(anchors), + ) + matched_idxs, matched_labels = self.anchor_matcher( + iou + ) # proposal_id -> highest_iou_gt_id, proposal_id -> [1 if iou > 0.7, 0 if iou < 0.3, -1 ow] + matched_labels = self._subsample_labels(matched_labels) + + all_pr_inds = torch.arange(len(anchors)).to(anchors.device) + + pos_pr_inds = all_pr_inds[matched_labels == 1] + pos_gt_inds = matched_idxs[pos_pr_inds] + pos_ious = iou[pos_gt_inds, pos_pr_inds] + pos_pr_inds, pos_gt_inds = self.postprocess_indices( + pos_pr_inds, pos_gt_inds, iou + ) + pos_pr_inds, pos_gt_inds = pos_pr_inds.to(anchors.device), pos_gt_inds.to( + anchors.device + ) + indices.append((pos_pr_inds, pos_gt_inds)) + return indices + + def postprocess_indices(self, pr_inds, gt_inds, iou): + return sample_topk_per_gt(pr_inds, gt_inds, iou, self.k) diff --git a/perception_models/apps/detection/DETA_pe/models/backbone.py b/perception_models/apps/detection/DETA_pe/models/backbone.py new file mode 100644 index 0000000000000000000000000000000000000000..a19daa47b339439ec67e34ef17e0b75533b3267b --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/models/backbone.py @@ -0,0 +1,235 @@ +# ------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------ +# Modified from DETR (https://github.com/facebookresearch/detr) +# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved +# ------------------------------------------------------------------------ + +""" +Backbone modules. +""" +from collections import OrderedDict +from functools import partial +from typing import Dict, List + +import torch +import torch.nn.functional as F +import torchvision +from torch import nn +from torch.cuda.amp import autocast +from torchvision.models._utils import IntermediateLayerGetter +from util.misc import is_main_process, NestedTensor + +from .position_encoding import build_position_encoding +from .swin import get_swinl +from .pev1 import get_pev1_and_fpn_backbone + + +class FrozenBatchNorm2d(torch.nn.Module): + """ + BatchNorm2d where the batch statistics and the affine parameters are fixed. + + Copy-paste from torchvision.misc.ops with added eps before rqsrt, + without which any other models than torchvision.models.resnet[18,34,50,101] + produce nans. + """ + + def __init__(self, n, eps=1e-5): + super(FrozenBatchNorm2d, self).__init__() + self.register_buffer("weight", torch.ones(n)) + self.register_buffer("bias", torch.zeros(n)) + self.register_buffer("running_mean", torch.zeros(n)) + self.register_buffer("running_var", torch.ones(n)) + self.eps = eps + + def _load_from_state_dict( + self, + state_dict, + prefix, + local_metadata, + strict, + missing_keys, + unexpected_keys, + error_msgs, + ): + num_batches_tracked_key = prefix + "num_batches_tracked" + if num_batches_tracked_key in state_dict: + del state_dict[num_batches_tracked_key] + + super(FrozenBatchNorm2d, self)._load_from_state_dict( + state_dict, + prefix, + local_metadata, + strict, + missing_keys, + unexpected_keys, + error_msgs, + ) + + def forward(self, x): + # move reshapes to the beginning + # to make it fuser-friendly + w = self.weight.reshape(1, -1, 1, 1) + b = self.bias.reshape(1, -1, 1, 1) + rv = self.running_var.reshape(1, -1, 1, 1) + rm = self.running_mean.reshape(1, -1, 1, 1) + eps = self.eps + scale = w * (rv + eps).rsqrt() + bias = b - rm * scale + return x * scale + bias + + +class BackboneBase(nn.Module): + + def __init__( + self, backbone: nn.Module, train_backbone: bool, return_interm_layers: bool + ): + super().__init__() + for name, parameter in backbone.named_parameters(): + if ( + not train_backbone + or "layer2" not in name + and "layer3" not in name + and "layer4" not in name + ): + parameter.requires_grad_(False) + if return_interm_layers: + # return_layers = {"layer1": "0", "layer2": "1", "layer3": "2", "layer4": "3"} + return_layers = {"layer2": "0", "layer3": "1", "layer4": "2"} + self.strides = [8, 16, 32] + self.num_channels = [512, 1024, 2048] + else: + return_layers = {"layer4": "0"} + self.strides = [32] + self.num_channels = [2048] + self.body = IntermediateLayerGetter(backbone, return_layers=return_layers) + + def forward(self, tensor_list: NestedTensor): + xs = self.body(tensor_list.tensors) + out: Dict[str, NestedTensor] = {} + for name, x in xs.items(): + m = tensor_list.mask + assert m is not None + mask = F.interpolate(m[None].float(), size=x.shape[-2:]).to(torch.bool)[0] + out[name] = NestedTensor(x, mask) + return out + + +class Backbone(BackboneBase): + """ResNet backbone with frozen BatchNorm.""" + + def __init__( + self, + name: str, + train_backbone: bool, + return_interm_layers: bool, + dilation: bool, + ): + norm_layer = FrozenBatchNorm2d + backbone = getattr(torchvision.models, name)( + replace_stride_with_dilation=[False, False, dilation], + pretrained=is_main_process(), + norm_layer=norm_layer, + ) + assert name not in ("resnet18", "resnet34"), "number of channels are hard coded" + super().__init__(backbone, train_backbone, return_interm_layers) + if dilation: + self.strides[-1] = self.strides[-1] // 2 + + +class SwinBackbone(nn.Module): + def __init__(self): + # we skip R50 FrozenBatchNorm2d, dilation, train l{2,3,4} only + super().__init__() + self.body = get_swinl() + self.features = ["res3", "res4", "res5"] + self.strides = [8, 16, 32] + self.num_channels = [384, 768, 1536] + + def forward(self, tensor_list: NestedTensor): + xs = self.body(tensor_list.tensors) + m = tensor_list.mask[None] + assert m is not None + out: Dict[str, NestedTensor] = {} + for name in self.features: + mask = F.interpolate(m.float(), size=xs[name].shape[-2:]).to(torch.bool)[0] + out[name] = NestedTensor(xs[name], mask) + return out + + +class PEv1Backbone(nn.Module): + def __init__(self, args): + super().__init__() + self.body = get_pev1_and_fpn_backbone(args) + self.features = self.body._out_features + + self.bf16 = args.bf16 + self.fp16 = args.fp16 + + _out_feature_strides = self.body._out_feature_strides + _out_feature_channels = self.body._out_feature_channels + self.strides = [_out_feature_strides[f] for f in _out_feature_strides.keys()] + self.num_channels = [ + _out_feature_channels[f] for f in _out_feature_channels.keys() + ] + + def forward(self, tensor_list: NestedTensor): + # xs = self.body(tensor_list.tensors) + # backbone + if self.bf16: + with autocast(dtype=torch.bfloat16): + xs = self.body(tensor_list.tensors.to(torch.bfloat16)) + xs = {k: v.float() for k, v in xs.items()} + elif self.fp16: + with autocast(dtype=torch.float16): + xs = self.body(tensor_list.tensors.half()) + xs = {k: v.float() for k, v in xs.items()} + else: + xs = self.body(tensor_list.tensors) + + m = tensor_list.mask[None] + assert m is not None + out: Dict[str, NestedTensor] = {} + + for name in self.features: + mask = F.interpolate(m.float(), size=xs[name].shape[-2:]).to(torch.bool)[0] + out[name] = NestedTensor(xs[name], mask) + return out + + +class Joiner(nn.Sequential): + def __init__(self, backbone, position_embedding): + super().__init__(backbone, position_embedding) + self.strides = backbone.strides + self.num_channels = backbone.num_channels + + def forward(self, tensor_list: NestedTensor): + xs = self[0](tensor_list) + out: List[NestedTensor] = [] + pos = [] + for name, x in sorted(xs.items()): + out.append(x) + + # position encoding + for x in out: + pos.append(self[1](x).to(x.tensors.dtype)) + + return out, pos + + +def build_backbone(args): + position_embedding = build_position_encoding(args) + train_backbone = args.lr_backbone > 0 + return_interm_layers = args.masks or (args.num_feature_levels > 1) + if "swin" in args.backbone: + backbone = SwinBackbone() + elif "pev1" in args.backbone: + backbone = PEv1Backbone(args) + else: + backbone = Backbone( + args.backbone, train_backbone, return_interm_layers, args.dilation + ) + model = Joiner(backbone, position_embedding) + return model diff --git a/perception_models/apps/detection/DETA_pe/models/deformable_detr.py b/perception_models/apps/detection/DETA_pe/models/deformable_detr.py new file mode 100644 index 0000000000000000000000000000000000000000..2f8cb9e2d74635b4938b1e7b7d2a1ce31e000094 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/models/deformable_detr.py @@ -0,0 +1,776 @@ +# ------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------ +# Modified from DETR (https://github.com/facebookresearch/detr) +# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved +# ------------------------------------------------------------------------ + +""" +Deformable DETR model and criterion classes. +""" +import copy +import math + +import torch +import torch.nn.functional as F +from torch import nn +from torchvision.ops.boxes import batched_nms + +from util import box_ops +from util.misc import ( + accuracy, + get_world_size, + interpolate, + inverse_sigmoid, + is_dist_avail_and_initialized, + nested_tensor_from_tensor_list, + NestedTensor, +) + +from .assigner import Stage1Assigner, Stage2Assigner + +from .backbone import build_backbone +from .deformable_transformer import build_deforamble_transformer +from .matcher import build_matcher +from .segmentation import ( + DETRsegm, + dice_loss, + PostProcessPanoptic, + PostProcessSegm, + sigmoid_focal_loss, +) +from .utils_fed_loss import get_fed_loss_inds, load_class_freq +from .utils_softnms import batched_soft_nms + + +def _get_clones(module, N): + return nn.ModuleList([copy.deepcopy(module) for i in range(N)]) + + +class DeformableDETR(nn.Module): + """This is the Deformable DETR module that performs object detection""" + + def __init__( + self, + backbone, + transformer, + num_classes, + num_queries, + num_feature_levels, + aux_loss=True, + with_box_refine=False, + two_stage=False, + ): + """Initializes the model. + Parameters: + backbone: torch module of the backbone to be used. See backbone.py + transformer: torch module of the transformer architecture. See transformer.py + num_classes: number of object classes + num_queries: number of object queries, ie detection slot. This is the maximal number of objects + DETR can detect in a single image. For COCO, we recommend 100 queries. + aux_loss: True if auxiliary decoding losses (loss at each decoder layer) are to be used. + with_box_refine: iterative bounding box refinement + two_stage: two-stage Deformable DETR + """ + super().__init__() + self.num_queries = num_queries + self.transformer = transformer + hidden_dim = transformer.d_model + self.class_embed = nn.Linear(hidden_dim, num_classes) + self.bbox_embed = MLP(hidden_dim, hidden_dim, 4, 3) + self.num_feature_levels = num_feature_levels + if not two_stage: + self.query_embed = nn.Embedding(num_queries, hidden_dim * 2) + if num_feature_levels > 1: + num_backbone_outs = len(backbone.strides) + input_proj_list = [] + for _ in range(num_backbone_outs): + in_channels = backbone.num_channels[_] + input_proj_list.append( + nn.Sequential( + nn.Conv2d(in_channels, hidden_dim, kernel_size=1), + nn.GroupNorm(32, hidden_dim), + ) + ) + for _ in range(num_feature_levels - num_backbone_outs): + input_proj_list.append( + nn.Sequential( + nn.Conv2d( + in_channels, hidden_dim, kernel_size=3, stride=2, padding=1 + ), + nn.GroupNorm(32, hidden_dim), + ) + ) + in_channels = hidden_dim + self.input_proj = nn.ModuleList(input_proj_list) + else: + self.input_proj = nn.ModuleList( + [ + nn.Sequential( + nn.Conv2d(backbone.num_channels[0], hidden_dim, kernel_size=1), + nn.GroupNorm(32, hidden_dim), + ) + ] + ) + self.backbone = backbone + self.aux_loss = aux_loss + self.with_box_refine = with_box_refine + self.two_stage = two_stage + + prior_prob = 0.01 + bias_value = -math.log((1 - prior_prob) / prior_prob) + self.class_embed.bias.data = torch.ones(num_classes) * bias_value + nn.init.constant_(self.bbox_embed.layers[-1].weight.data, 0) + nn.init.constant_(self.bbox_embed.layers[-1].bias.data, 0) + for proj in self.input_proj: + nn.init.xavier_uniform_(proj[0].weight, gain=1) + nn.init.constant_(proj[0].bias, 0) + + # if two-stage, the last class_embed and bbox_embed is for region proposal generation + num_pred = ( + (transformer.decoder.num_layers + 1) + if two_stage + else transformer.decoder.num_layers + ) + if with_box_refine: + self.class_embed = _get_clones(self.class_embed, num_pred) + self.bbox_embed = _get_clones(self.bbox_embed, num_pred) + nn.init.constant_(self.bbox_embed[0].layers[-1].bias.data[2:], -2.0) + # hack implementation for iterative bounding box refinement + self.transformer.decoder.bbox_embed = self.bbox_embed + else: + nn.init.constant_(self.bbox_embed.layers[-1].bias.data[2:], -2.0) + self.class_embed = nn.ModuleList( + [self.class_embed for _ in range(num_pred)] + ) + self.bbox_embed = nn.ModuleList([self.bbox_embed for _ in range(num_pred)]) + self.transformer.decoder.bbox_embed = None + if two_stage: + # hack implementation for two-stage + self.transformer.decoder.class_embed = self.class_embed + for box_embed in self.bbox_embed: + nn.init.constant_(box_embed.layers[-1].bias.data[2:], 0.0) + + def forward(self, samples: NestedTensor): + """The forward expects a NestedTensor, which consists of: + - samples.tensor: batched images, of shape [batch_size x 3 x H x W] + - samples.mask: a binary mask of shape [batch_size x H x W], containing 1 on padded pixels + + It returns a dict with the following elements: + - "pred_logits": the classification logits (including no-object) for all queries. + Shape= [batch_size x num_queries x (num_classes + 1)] + - "pred_boxes": The normalized boxes coordinates for all queries, represented as + (center_x, center_y, height, width). These values are normalized in [0, 1], + relative to the size of each individual image (disregarding possible padding). + See PostProcess for information on how to retrieve the unnormalized bounding box. + - "aux_outputs": Optional, only returned when auxilary losses are activated. It is a list of + dictionnaries containing the two above keys for each decoder layer. + """ + if not isinstance(samples, NestedTensor): + samples = nested_tensor_from_tensor_list(samples) + features, pos = self.backbone(samples) + + srcs = [] + masks = [] + for l, feat in enumerate(features): + src, mask = feat.decompose() + srcs.append(self.input_proj[l](src)) + masks.append(mask) + assert mask is not None + if self.num_feature_levels > len(srcs): + _len_srcs = len(srcs) + for l in range(_len_srcs, self.num_feature_levels): + if l == _len_srcs: + src = self.input_proj[l](features[-1].tensors) + else: + src = self.input_proj[l](srcs[-1]) + m = samples.mask + mask = F.interpolate(m[None].float(), size=src.shape[-2:]).to( + torch.bool + )[0] + pos_l = self.backbone[1](NestedTensor(src, mask)).to(src.dtype) + srcs.append(src) + masks.append(mask) + pos.append(pos_l) + + query_embeds = None + if not self.two_stage: + query_embeds = self.query_embed.weight + ( + hs, + init_reference, + inter_references, + enc_outputs_class, + enc_outputs_coord_unact, + anchors, + ) = self.transformer(srcs, masks, pos, query_embeds) + + outputs_classes = [] + outputs_coords = [] + for lvl in range(hs.shape[0]): + if lvl == 0: + reference = init_reference + else: + reference = inter_references[lvl - 1] + reference = inverse_sigmoid(reference) + outputs_class = self.class_embed[lvl](hs[lvl]) + tmp = self.bbox_embed[lvl](hs[lvl]) + if reference.shape[-1] == 4: + tmp += reference + else: + assert reference.shape[-1] == 2 + tmp[..., :2] += reference + outputs_coord = tmp.sigmoid() + outputs_classes.append(outputs_class) + outputs_coords.append(outputs_coord) + outputs_class = torch.stack(outputs_classes) + outputs_coord = torch.stack(outputs_coords) + + out = { + "pred_logits": outputs_class[-1], + "pred_boxes": outputs_coord[-1], + "init_reference": init_reference, + } + if self.aux_loss: + out["aux_outputs"] = self._set_aux_loss(outputs_class, outputs_coord) + + if self.two_stage: + enc_outputs_coord = enc_outputs_coord_unact.sigmoid() + out["enc_outputs"] = { + "pred_logits": enc_outputs_class, + "pred_boxes": enc_outputs_coord, + "anchors": anchors, + } + return out + + @torch.jit.unused + def _set_aux_loss(self, outputs_class, outputs_coord): + # this is a workaround to make torchscript happy, as torchscript + # doesn't support dictionary with non-homogeneous values, such + # as a dict having both a Tensor and a list. + return [ + {"pred_logits": a, "pred_boxes": b} + for a, b in zip(outputs_class[:-1], outputs_coord[:-1]) + ] + + +class SetCriterion(nn.Module): + """This class computes the loss for DETR. + The process happens in two steps: + 1) we compute hungarian assignment between ground truth boxes and the outputs of the model + 2) we supervise each pair of matched ground-truth / prediction (supervise class and box) + """ + + def __init__( + self, + num_classes, + matcher, + weight_dict, + losses, + focal_alpha=0.25, + num_queries=300, + assign_first_stage=False, + assign_second_stage=False, + use_fed_loss=False, + ): + """Create the criterion. + Parameters: + num_classes: number of object categories, omitting the special no-object category + matcher: module able to compute a matching between targets and proposals + weight_dict: dict containing as key the names of the losses and as values their relative weight. + losses: list of all the losses to be applied. See get_loss for list of available losses. + focal_alpha: alpha in Focal Loss + """ + super().__init__() + self.num_classes = num_classes + self.matcher = matcher + self.weight_dict = weight_dict + self.losses = losses + self.focal_alpha = focal_alpha + self.assign_first_stage = assign_first_stage + self.assign_second_stage = assign_second_stage + + if self.assign_first_stage: + self.stg1_assigner = Stage1Assigner() + if self.assign_second_stage: + self.stg2_assigner = Stage2Assigner(num_queries) + + self.use_fed_loss = use_fed_loss + if self.use_fed_loss: + print("Using federated loss") + print("Using federated loss") + print("Using federated loss") + self.register_buffer("fed_loss_weight", load_class_freq(freq_weight=0.5)) + + def loss_labels(self, outputs, targets, indices, num_boxes, log=True): + """Classification loss (NLL) + targets dicts must contain the key "labels" containing a tensor of dim [nb_target_boxes] + """ + assert "pred_logits" in outputs + src_logits = outputs["pred_logits"] + + idx = self._get_src_permutation_idx(indices) + target_classes_o = torch.cat( + [t["labels"][J] for t, (_, J) in zip(targets, indices)] + ) + target_classes = torch.full( + src_logits.shape[:2], + self.num_classes, + dtype=torch.int64, + device=src_logits.device, + ) + target_classes[idx] = target_classes_o + + target_classes_onehot = torch.zeros( + [src_logits.shape[0], src_logits.shape[1], src_logits.shape[2] + 1], + dtype=src_logits.dtype, + layout=src_logits.layout, + device=src_logits.device, + ) + target_classes_onehot.scatter_(2, target_classes.unsqueeze(-1), 1) + + target_classes_onehot = target_classes_onehot[:, :, :-1] + if self.use_fed_loss: + inds = ( + get_fed_loss_inds( + gt_classes=target_classes_o - 1, + num_sample_cats=50, + weight=self.fed_loss_weight, + C=target_classes_onehot.shape[2] - 1, + ) + + 1 + ) # pay attention to the -1 and +1 + loss_ce = ( + sigmoid_focal_loss( + src_logits[:, :, inds], + target_classes_onehot[:, :, inds], + num_boxes, + alpha=self.focal_alpha, + gamma=2, + ) + * src_logits.shape[1] + ) + else: + loss_ce = ( + sigmoid_focal_loss( + src_logits, + target_classes_onehot, + num_boxes, + alpha=self.focal_alpha, + gamma=2, + ) + * src_logits.shape[1] + ) + losses = {"loss_ce": loss_ce} + + if log: + # TODO this should probably be a separate loss, not hacked in this one here + losses["class_error"] = 100 - accuracy(src_logits[idx], target_classes_o)[0] + return losses + + @torch.no_grad() + def loss_cardinality(self, outputs, targets, indices, num_boxes): + """Compute the cardinality error, ie the absolute error in the number of predicted non-empty boxes + This is not really a loss, it is intended for logging purposes only. It doesn't propagate gradients + """ + pred_logits = outputs["pred_logits"] + device = pred_logits.device + tgt_lengths = torch.as_tensor( + [len(v["labels"]) for v in targets], device=device + ) + # Count the number of predictions that are NOT "no-object" (which is the last class) + card_pred = (pred_logits.argmax(-1) != pred_logits.shape[-1] - 1).sum(1) + card_err = F.l1_loss(card_pred.float(), tgt_lengths.float()) + losses = {"cardinality_error": card_err} + return losses + + def loss_boxes(self, outputs, targets, indices, num_boxes): + """Compute the losses related to the bounding boxes, the L1 regression loss and the GIoU loss + targets dicts must contain the key "boxes" containing a tensor of dim [nb_target_boxes, 4] + The target boxes are expected in format (center_x, center_y, h, w), normalized by the image size. + """ + assert "pred_boxes" in outputs + idx = self._get_src_permutation_idx(indices) + src_boxes = outputs["pred_boxes"][idx] + target_boxes = torch.cat( + [t["boxes"][i] for t, (_, i) in zip(targets, indices)], dim=0 + ) + + loss_bbox = F.l1_loss(src_boxes, target_boxes, reduction="none") + + losses = {} + losses["loss_bbox"] = loss_bbox.sum() / num_boxes + + loss_giou = 1 - torch.diag( + box_ops.generalized_box_iou( + box_ops.box_cxcywh_to_xyxy(src_boxes), + box_ops.box_cxcywh_to_xyxy(target_boxes), + ) + ) + losses["loss_giou"] = loss_giou.sum() / num_boxes + return losses + + def loss_masks(self, outputs, targets, indices, num_boxes): + """Compute the losses related to the masks: the focal loss and the dice loss. + targets dicts must contain the key "masks" containing a tensor of dim [nb_target_boxes, h, w] + """ + assert "pred_masks" in outputs + + src_idx = self._get_src_permutation_idx(indices) + tgt_idx = self._get_tgt_permutation_idx(indices) + + src_masks = outputs["pred_masks"] + + # TODO use valid to mask invalid areas due to padding in loss + target_masks, valid = nested_tensor_from_tensor_list( + [t["masks"] for t in targets] + ).decompose() + target_masks = target_masks.to(src_masks) + + src_masks = src_masks[src_idx] + # upsample predictions to the target size + src_masks = interpolate( + src_masks[:, None], + size=target_masks.shape[-2:], + mode="bilinear", + align_corners=False, + ) + src_masks = src_masks[:, 0].flatten(1) + + target_masks = target_masks[tgt_idx].flatten(1) + + losses = { + "loss_mask": sigmoid_focal_loss(src_masks, target_masks, num_boxes), + "loss_dice": dice_loss(src_masks, target_masks, num_boxes), + } + return losses + + def _get_src_permutation_idx(self, indices): + # permute predictions following indices + batch_idx = torch.cat( + [torch.full_like(src, i) for i, (src, _) in enumerate(indices)] + ) + src_idx = torch.cat([src for (src, _) in indices]) + return batch_idx, src_idx + + def _get_tgt_permutation_idx(self, indices): + # permute targets following indices + batch_idx = torch.cat( + [torch.full_like(tgt, i) for i, (_, tgt) in enumerate(indices)] + ) + tgt_idx = torch.cat([tgt for (_, tgt) in indices]) + return batch_idx, tgt_idx + + def get_loss(self, loss, outputs, targets, indices, num_boxes, **kwargs): + loss_map = { + "labels": self.loss_labels, + "cardinality": self.loss_cardinality, + "boxes": self.loss_boxes, + "masks": self.loss_masks, + } + assert loss in loss_map, f"do you really want to compute {loss} loss?" + return loss_map[loss](outputs, targets, indices, num_boxes, **kwargs) + + def forward(self, outputs, targets): + """This performs the loss computation. + Parameters: + outputs: dict of tensors, see the output specification of the model for the format + targets: list of dicts, such that len(targets) == batch_size. + The expected keys in each dict depends on the losses applied, see each loss' doc + """ + outputs_without_aux = { + k: v + for k, v in outputs.items() + if k != "aux_outputs" and k != "enc_outputs" + } + + # Retrieve the matching between the outputs of the last layer and the targets + if self.assign_second_stage: + indices = self.stg2_assigner(outputs_without_aux, targets) + else: + indices = self.matcher(outputs_without_aux, targets) + + # Compute the average number of target boxes accross all nodes, for normalization purposes + num_boxes = sum(len(t["labels"]) for t in targets) + num_boxes = torch.as_tensor( + [num_boxes], dtype=torch.float, device=next(iter(outputs.values())).device + ) + if is_dist_avail_and_initialized(): + torch.distributed.all_reduce(num_boxes) + num_boxes = torch.clamp(num_boxes / get_world_size(), min=1).item() + + # Compute all the requested losses + losses = {} + for loss in self.losses: + kwargs = {} + losses.update( + self.get_loss(loss, outputs, targets, indices, num_boxes, **kwargs) + ) + + # In case of auxiliary losses, we repeat this process with the output of each intermediate layer. + if "aux_outputs" in outputs: + for i, aux_outputs in enumerate(outputs["aux_outputs"]): + if not self.assign_second_stage: + indices = self.matcher(aux_outputs, targets) + for loss in self.losses: + if loss == "masks": + # Intermediate masks losses are too costly to compute, we ignore them. + continue + kwargs = {} + if loss == "labels": + # Logging is enabled only for the last layer + kwargs["log"] = False + l_dict = self.get_loss( + loss, aux_outputs, targets, indices, num_boxes, **kwargs + ) + l_dict = {k + f"_{i}": v for k, v in l_dict.items()} + losses.update(l_dict) + + if "enc_outputs" in outputs: + enc_outputs = outputs["enc_outputs"] + bin_targets = copy.deepcopy(targets) + for bt in bin_targets: + bt["labels"] = torch.zeros_like(bt["labels"]) + if self.assign_first_stage: + indices = self.stg1_assigner(enc_outputs, bin_targets) + else: + indices = self.matcher(enc_outputs, bin_targets) + for loss in self.losses: + if loss == "masks": + # Intermediate masks losses are too costly to compute, we ignore them. + continue + kwargs = {} + if loss == "labels": + # Logging is enabled only for the last layer + kwargs["log"] = False + l_dict = self.get_loss( + loss, enc_outputs, bin_targets, indices, num_boxes, **kwargs + ) + l_dict = {k + f"_enc": v for k, v in l_dict.items()} + losses.update(l_dict) + + return losses + + +class PostProcess(nn.Module): + """This module converts the model's output into the format expected by the coco api""" + + @torch.no_grad() + def forward(self, outputs, target_sizes, num_topk=100): + """Perform the computation + Parameters: + outputs: raw outputs of the model + target_sizes: tensor of dimension [batch_size x 2] containing the size of each images of the batch + For evaluation, this must be the original image size (before any data augmentation) + For visualization, this should be the image size after data augment, but before padding + """ + out_logits, out_bbox = outputs["pred_logits"], outputs["pred_boxes"] + + assert len(out_logits) == len(target_sizes) + assert target_sizes.shape[1] == 2 + + prob = out_logits.sigmoid() + topk_values, topk_indexes = torch.topk( + prob.view(out_logits.shape[0], -1), num_topk, dim=1 + ) + scores = topk_values + topk_boxes = topk_indexes // out_logits.shape[2] + labels = topk_indexes % out_logits.shape[2] + boxes = box_ops.box_cxcywh_to_xyxy(out_bbox) + boxes = torch.gather(boxes, 1, topk_boxes.unsqueeze(-1).repeat(1, 1, 4)) + + # and from relative [0, 1] to absolute [0, height] coordinates + img_h, img_w = target_sizes.unbind(1) + scale_fct = torch.stack([img_w, img_h, img_w, img_h], dim=1) + boxes = boxes * scale_fct[:, None, :] + + results = [ + {"scores": s, "labels": l, "boxes": b} + for s, l, b in zip(scores, labels, boxes) + ] + + return results + + +class NMSPostProcess(nn.Module): + """This module converts the model's output into the format expected by the coco api""" + + @torch.no_grad() + def forward( + self, + outputs, + target_sizes, + num_topk=100, + soft_nms=False, + nms_thresh=0.7, + method="quad", + quad_scale=1.0, + ): + """Perform the computation + Parameters: + outputs: raw outputs of the model + target_sizes: tensor of dimension [batch_size x 2] containing the size of each images of the batch + For evaluation, this must be the original image size (before any data augmentation) + For visualization, this should be the image size after data augment, but before padding + """ + out_logits, out_bbox = outputs["pred_logits"], outputs["pred_boxes"] + bs, n_queries, n_cls = out_logits.shape + + assert len(out_logits) == len(target_sizes) + assert target_sizes.shape[1] == 2 + + prob = out_logits.sigmoid() + + all_scores = prob.view(bs, n_queries * n_cls).to(out_logits.device) + all_indexes = ( + torch.arange(n_queries * n_cls)[None].repeat(bs, 1).to(out_logits.device) + ) + all_boxes = all_indexes // out_logits.shape[2] + all_labels = all_indexes % out_logits.shape[2] + + boxes = box_ops.box_cxcywh_to_xyxy(out_bbox) + boxes = torch.gather(boxes, 1, all_boxes.unsqueeze(-1).repeat(1, 1, 4)) + + # and from relative [0, 1] to absolute [0, height] coordinates + img_h, img_w = target_sizes.unbind(1) + scale_fct = torch.stack([img_w, img_h, img_w, img_h], dim=1) + boxes = boxes * scale_fct[:, None, :] + + results = [] + for b in range(bs): + box = boxes[b] + score = all_scores[b] + lbls = all_labels[b] + + if soft_nms: + if n_queries * n_cls > 2000: + pre_topk = score.topk(2000).indices + box = box[pre_topk] + score = score[pre_topk] + lbls = lbls[pre_topk] + # Apply soft-NMS to get indices and updated scores + keep_inds, updated_scores = batched_soft_nms( + box, + score, + lbls, + nms_thresh, + method=method, + quad_scale=quad_scale, + )[:num_topk] + + results.append( + { + "scores": updated_scores, + "labels": lbls[keep_inds], + "boxes": box[keep_inds], + } + ) + else: + if n_queries * n_cls > 10000: + pre_topk = score.topk(10000).indices + box = box[pre_topk] + score = score[pre_topk] + lbls = lbls[pre_topk] + keep_inds = batched_nms(box, score, lbls, nms_thresh)[:num_topk] + results.append( + { + "scores": score[keep_inds], + "labels": lbls[keep_inds], + "boxes": box[keep_inds], + } + ) + + return results + + +class MLP(nn.Module): + """Very simple multi-layer perceptron (also called FFN)""" + + def __init__(self, input_dim, hidden_dim, output_dim, num_layers): + super().__init__() + self.num_layers = num_layers + h = [hidden_dim] * (num_layers - 1) + self.layers = nn.ModuleList( + nn.Linear(n, k) for n, k in zip([input_dim] + h, h + [output_dim]) + ) + + def forward(self, x): + for i, layer in enumerate(self.layers): + x = F.relu(layer(x)) if i < self.num_layers - 1 else layer(x) + return x + + +def build(args): + # num_classes = 20 if args.dataset_file != 'coco' else 91 + if args.dataset_file == "coco_panoptic": + num_classes = 250 + elif args.dataset_file == "voc": + num_classes = 20 + elif args.dataset_file == "objects365": + num_classes = 366 + elif args.dataset_file == "lvis": + num_classes = 1204 + else: # coco + num_classes = 91 + device = torch.device(args.device) + + backbone = build_backbone(args) + + transformer = build_deforamble_transformer(args) + model = DeformableDETR( + backbone, + transformer, + num_classes=num_classes, + num_queries=args.num_queries, + num_feature_levels=args.num_feature_levels, + aux_loss=args.aux_loss, + with_box_refine=args.with_box_refine, + two_stage=args.two_stage, + ) + if args.masks: + model = DETRsegm(model, freeze_detr=(args.frozen_weights is not None)) + matcher = build_matcher(args) + weight_dict = {"loss_ce": args.cls_loss_coef, "loss_bbox": args.bbox_loss_coef} + weight_dict["loss_giou"] = args.giou_loss_coef + if args.masks: + weight_dict["loss_mask"] = args.mask_loss_coef + weight_dict["loss_dice"] = args.dice_loss_coef + # TODO this is a hack + if args.aux_loss: + aux_weight_dict = {} + for i in range(args.dec_layers - 1): + aux_weight_dict.update({k + f"_{i}": v for k, v in weight_dict.items()}) + aux_weight_dict.update({k + f"_enc": v for k, v in weight_dict.items()}) + weight_dict.update(aux_weight_dict) + + losses = ["labels", "boxes", "cardinality"] + if args.masks: + losses += ["masks"] + # num_classes, matcher, weight_dict, losses, focal_alpha=0.25 + criterion = SetCriterion( + num_classes, + matcher, + weight_dict, + losses, + focal_alpha=args.focal_alpha, + num_queries=args.num_queries, + assign_first_stage=args.assign_first_stage, + assign_second_stage=args.assign_second_stage, + use_fed_loss=args.use_fed_loss, + ) + criterion.to(device) + if args.assign_second_stage: + postprocessors = {"bbox": NMSPostProcess()} + else: + postprocessors = {"bbox": PostProcess()} + if args.masks: + postprocessors["segm"] = PostProcessSegm() + if args.dataset_file == "coco_panoptic": + is_thing_map = {i: i <= 90 for i in range(201)} + postprocessors["panoptic"] = PostProcessPanoptic( + is_thing_map, threshold=0.85 + ) + + return model, criterion, postprocessors diff --git a/perception_models/apps/detection/DETA_pe/models/deformable_transformer.py b/perception_models/apps/detection/DETA_pe/models/deformable_transformer.py new file mode 100644 index 0000000000000000000000000000000000000000..e4cfd859579766cf3c1d6dfe4ef24634fb7e3e79 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/models/deformable_transformer.py @@ -0,0 +1,451 @@ +# ------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------ +# Modified from DETR (https://github.com/facebookresearch/detr) +# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved +# ------------------------------------------------------------------------ + +import copy +from typing import Optional, List +import math + +import torch +import torch.nn.functional as F +from torch import nn, Tensor +from torch.nn.init import xavier_uniform_, constant_, uniform_, normal_ + +from util.misc import inverse_sigmoid +from models.ops.modules import MSDeformAttn + +from torchvision.ops.boxes import batched_nms +from util.box_ops import box_cxcywh_to_xyxy + +class DeformableTransformer(nn.Module): + def __init__(self, d_model=256, nhead=8, + num_encoder_layers=6, num_decoder_layers=6, dim_feedforward=1024, dropout=0.1, + activation="relu", return_intermediate_dec=False, + num_feature_levels=4, dec_n_points=4, enc_n_points=4, + two_stage=False, two_stage_num_proposals=300, + assign_first_stage=False): + super().__init__() + + self.d_model = d_model + self.nhead = nhead + self.two_stage = two_stage + self.two_stage_num_proposals = two_stage_num_proposals + self.assign_first_stage = assign_first_stage + + encoder_layer = DeformableTransformerEncoderLayer(d_model, dim_feedforward, + dropout, activation, + num_feature_levels, nhead, enc_n_points) + self.encoder = DeformableTransformerEncoder(encoder_layer, num_encoder_layers) + + decoder_layer = DeformableTransformerDecoderLayer(d_model, dim_feedforward, + dropout, activation, + num_feature_levels, nhead, dec_n_points) + self.decoder = DeformableTransformerDecoder(decoder_layer, num_decoder_layers, return_intermediate_dec) + + self.level_embed = nn.Parameter(torch.Tensor(num_feature_levels, d_model)) + + if two_stage: + self.enc_output = nn.Linear(d_model, d_model) + self.enc_output_norm = nn.LayerNorm(d_model) + self.pos_trans = nn.Linear(d_model * 2, d_model * 2) + self.pos_trans_norm = nn.LayerNorm(d_model * 2) + self.pix_trans = nn.Linear(d_model, d_model) + self.pix_trans_norm = nn.LayerNorm(d_model) + else: + self.reference_points = nn.Linear(d_model, 2) + + self._reset_parameters() + + def _reset_parameters(self): + for p in self.parameters(): + if p.dim() > 1: + nn.init.xavier_uniform_(p) + for m in self.modules(): + if isinstance(m, MSDeformAttn): + m._reset_parameters() + if not self.two_stage: + xavier_uniform_(self.reference_points.weight.data, gain=1.0) + constant_(self.reference_points.bias.data, 0.) + normal_(self.level_embed) + + def get_proposal_pos_embed(self, proposals): + num_pos_feats = 128 + temperature = 10000 + scale = 2 * math.pi + + dim_t = torch.arange(num_pos_feats, dtype=torch.float32, device=proposals.device) + dim_t = temperature ** (2 * (dim_t // 2) / num_pos_feats) + # N, L, 4 + proposals = proposals.sigmoid() * scale + # N, L, 4, 128 + pos = proposals[:, :, :, None] / dim_t + # N, L, 4, 64, 2 + pos = torch.stack((pos[:, :, :, 0::2].sin(), pos[:, :, :, 1::2].cos()), dim=4).flatten(2) + return pos + + def gen_encoder_output_proposals(self, memory, memory_padding_mask, spatial_shapes): + N_, S_, C_ = memory.shape + base_scale = 4.0 + proposals = [] + _cur = 0 + level_ids = [] + for lvl, (H_, W_) in enumerate(spatial_shapes): + mask_flatten_ = memory_padding_mask[:, _cur:(_cur + H_ * W_)].view(N_, H_, W_, 1) + valid_H = torch.sum(~mask_flatten_[:, :, 0, 0], 1) + valid_W = torch.sum(~mask_flatten_[:, 0, :, 0], 1) + + grid_y, grid_x = torch.meshgrid(torch.linspace(0, H_ - 1, H_, dtype=torch.float32, device=memory.device), + torch.linspace(0, W_ - 1, W_, dtype=torch.float32, device=memory.device)) + grid = torch.cat([grid_x.unsqueeze(-1), grid_y.unsqueeze(-1)], -1) + + scale = torch.cat([valid_W.unsqueeze(-1), valid_H.unsqueeze(-1)], 1).view(N_, 1, 1, 2) + grid = (grid.unsqueeze(0).expand(N_, -1, -1, -1) + 0.5) / scale + wh = torch.ones_like(grid) * 0.05 * (2.0 ** lvl) + proposal = torch.cat((grid, wh), -1).view(N_, -1, 4) + proposals.append(proposal) + _cur += (H_ * W_) + level_ids.append(grid.new_ones(H_ * W_, dtype=torch.long) * lvl) + output_proposals = torch.cat(proposals, 1) + output_proposals_valid = ((output_proposals > 0.01) & (output_proposals < 0.99)).all(-1, keepdim=True) + output_proposals = torch.log(output_proposals / (1 - output_proposals)) + output_proposals = output_proposals.masked_fill(memory_padding_mask.unsqueeze(-1), float('inf')) + output_proposals = output_proposals.masked_fill(~output_proposals_valid, float('inf')) + + output_memory = memory + output_memory = output_memory.masked_fill(memory_padding_mask.unsqueeze(-1), float(0)) + output_memory = output_memory.masked_fill(~output_proposals_valid, float(0)) + output_memory = self.enc_output_norm(self.enc_output(output_memory)) + level_ids = torch.cat(level_ids) + return output_memory, output_proposals, level_ids + + def get_valid_ratio(self, mask): + _, H, W = mask.shape + valid_H = torch.sum(~mask[:, :, 0], 1) + valid_W = torch.sum(~mask[:, 0, :], 1) + valid_ratio_h = valid_H.float() / H + valid_ratio_w = valid_W.float() / W + valid_ratio = torch.stack([valid_ratio_w, valid_ratio_h], -1) + return valid_ratio + + def forward(self, srcs, masks, pos_embeds, query_embed=None): + assert self.two_stage or query_embed is not None + + # prepare input for encoder + src_flatten = [] + mask_flatten = [] + lvl_pos_embed_flatten = [] + spatial_shapes = [] + for lvl, (src, mask, pos_embed) in enumerate(zip(srcs, masks, pos_embeds)): + bs, c, h, w = src.shape + spatial_shape = (h, w) + spatial_shapes.append(spatial_shape) + src = src.flatten(2).transpose(1, 2) + mask = mask.flatten(1) + pos_embed = pos_embed.flatten(2).transpose(1, 2) + lvl_pos_embed = pos_embed + self.level_embed[lvl].view(1, 1, -1) + lvl_pos_embed_flatten.append(lvl_pos_embed) + src_flatten.append(src) + mask_flatten.append(mask) + src_flatten = torch.cat(src_flatten, 1) + mask_flatten = torch.cat(mask_flatten, 1) + lvl_pos_embed_flatten = torch.cat(lvl_pos_embed_flatten, 1) + spatial_shapes = torch.as_tensor(spatial_shapes, dtype=torch.long, device=src_flatten.device) + level_start_index = torch.cat((spatial_shapes.new_zeros((1, )), spatial_shapes.prod(1).cumsum(0)[:-1])) + valid_ratios = torch.stack([self.get_valid_ratio(m) for m in masks], 1) + + # encoder + memory = self.encoder(src_flatten, spatial_shapes, level_start_index, valid_ratios, lvl_pos_embed_flatten, mask_flatten) + + # prepare input for decoder + bs, _, c = memory.shape + if self.two_stage: + output_memory, output_proposals, level_ids = self.gen_encoder_output_proposals(memory, mask_flatten, spatial_shapes) + + # hack implementation for two-stage Deformable DETR + enc_outputs_class = self.decoder.class_embed[self.decoder.num_layers](output_memory) + enc_outputs_coord_unact = self.decoder.bbox_embed[self.decoder.num_layers](output_memory) + output_proposals + + topk = self.two_stage_num_proposals + proposal_logit = enc_outputs_class[..., 0] + + if self.assign_first_stage: + proposal_boxes = box_cxcywh_to_xyxy(enc_outputs_coord_unact.sigmoid().float()).clamp(0, 1) + topk_proposals = [] + for b in range(bs): + prop_boxes_b = proposal_boxes[b] + prop_logits_b = proposal_logit[b] + + # pre-nms per-level topk + pre_nms_topk = 1000 + pre_nms_inds = [] + for lvl in range(len(spatial_shapes)): + lvl_mask = level_ids == lvl + pre_nms_inds.append(torch.topk(prop_logits_b.sigmoid() * lvl_mask, pre_nms_topk)[1]) + pre_nms_inds = torch.cat(pre_nms_inds) + + # nms on topk indices + post_nms_inds = batched_nms(prop_boxes_b[pre_nms_inds], prop_logits_b[pre_nms_inds], level_ids[pre_nms_inds], 0.9) + keep_inds = pre_nms_inds[post_nms_inds] + + if len(keep_inds) < self.two_stage_num_proposals: + print(f'[WARNING] nms proposals ({len(keep_inds)}) < {self.two_stage_num_proposals}, running naive topk') + keep_inds = torch.topk(proposal_logit[b], topk)[1] + + # keep top Q/L indices for L levels + q_per_l = topk // len(spatial_shapes) + is_level_ordered = level_ids[keep_inds][None] == torch.arange(len(spatial_shapes), device=level_ids.device)[:,None] # LS + keep_inds_mask = is_level_ordered & (is_level_ordered.cumsum(1) <= q_per_l) # LS + keep_inds_mask = keep_inds_mask.any(0) # S + + # pad to Q indices (might let ones filtered from pre-nms sneak by... unlikely because we pick high conf anyways) + if keep_inds_mask.sum() < topk: + num_to_add = topk - keep_inds_mask.sum() + pad_inds = (~keep_inds_mask).nonzero()[:num_to_add] + keep_inds_mask[pad_inds] = True + + # index + keep_inds_topk = keep_inds[keep_inds_mask] + topk_proposals.append(keep_inds_topk) + topk_proposals = torch.stack(topk_proposals) + else: + topk_proposals = torch.topk(proposal_logit, topk, dim=1)[1] + + topk_coords_unact = torch.gather(enc_outputs_coord_unact, 1, topk_proposals.unsqueeze(-1).repeat(1, 1, 4)) + topk_coords_unact = topk_coords_unact.detach() + reference_points = topk_coords_unact.sigmoid() + init_reference_out = reference_points + pos_trans_out = self.pos_trans_norm(self.pos_trans(self.get_proposal_pos_embed(topk_coords_unact))) + query_embed, tgt = torch.split(pos_trans_out, c, dim=2) + + topk_feats = torch.stack([output_memory[b][topk_proposals[b]] for b in range(bs)]).detach() + tgt = tgt + self.pix_trans_norm(self.pix_trans(topk_feats)) + else: + query_embed, tgt = torch.split(query_embed, c, dim=1) + query_embed = query_embed.unsqueeze(0).expand(bs, -1, -1) + tgt = tgt.unsqueeze(0).expand(bs, -1, -1) + reference_points = self.reference_points(query_embed).sigmoid() + init_reference_out = reference_points + + # decoder + hs, inter_references = self.decoder(tgt, reference_points, memory, + spatial_shapes, level_start_index, valid_ratios, query_embed, mask_flatten) + + inter_references_out = inter_references + if self.two_stage: + return hs, init_reference_out, inter_references_out, enc_outputs_class, enc_outputs_coord_unact, output_proposals.sigmoid() + return hs, init_reference_out, inter_references_out, None, None, None + + +class DeformableTransformerEncoderLayer(nn.Module): + def __init__(self, + d_model=256, d_ffn=1024, + dropout=0.1, activation="relu", + n_levels=4, n_heads=8, n_points=4): + super().__init__() + + # self attention + self.self_attn = MSDeformAttn(d_model, n_levels, n_heads, n_points) + self.dropout1 = nn.Dropout(dropout) + self.norm1 = nn.LayerNorm(d_model) + + # ffn + self.linear1 = nn.Linear(d_model, d_ffn) + self.activation = _get_activation_fn(activation) + self.dropout2 = nn.Dropout(dropout) + self.linear2 = nn.Linear(d_ffn, d_model) + self.dropout3 = nn.Dropout(dropout) + self.norm2 = nn.LayerNorm(d_model) + + @staticmethod + def with_pos_embed(tensor, pos): + return tensor if pos is None else tensor + pos + + def forward_ffn(self, src): + src2 = self.linear2(self.dropout2(self.activation(self.linear1(src)))) + src = src + self.dropout3(src2) + src = self.norm2(src) + return src + + def forward(self, src, pos, reference_points, spatial_shapes, level_start_index, padding_mask=None): + # self attention + src2 = self.self_attn(self.with_pos_embed(src, pos), reference_points, src, spatial_shapes, level_start_index, padding_mask) + src = src + self.dropout1(src2) + src = self.norm1(src) + + # ffn + src = self.forward_ffn(src) + + return src + + +class DeformableTransformerEncoder(nn.Module): + def __init__(self, encoder_layer, num_layers): + super().__init__() + self.layers = _get_clones(encoder_layer, num_layers) + self.num_layers = num_layers + + @staticmethod + def get_reference_points(spatial_shapes, valid_ratios, device): + reference_points_list = [] + for lvl, (H_, W_) in enumerate(spatial_shapes): + + ref_y, ref_x = torch.meshgrid(torch.linspace(0.5, H_ - 0.5, H_, dtype=torch.float32, device=device), + torch.linspace(0.5, W_ - 0.5, W_, dtype=torch.float32, device=device)) + ref_y = ref_y.reshape(-1)[None] / (valid_ratios[:, None, lvl, 1] * H_) + ref_x = ref_x.reshape(-1)[None] / (valid_ratios[:, None, lvl, 0] * W_) + ref = torch.stack((ref_x, ref_y), -1) + reference_points_list.append(ref) + reference_points = torch.cat(reference_points_list, 1) + reference_points = reference_points[:, :, None] * valid_ratios[:, None] + return reference_points + + def forward(self, src, spatial_shapes, level_start_index, valid_ratios, pos=None, padding_mask=None): + output = src + reference_points = self.get_reference_points(spatial_shapes, valid_ratios, device=src.device) + for _, layer in enumerate(self.layers): + output = layer(output, pos, reference_points, spatial_shapes, level_start_index, padding_mask) + + return output + + +class DeformableTransformerDecoderLayer(nn.Module): + def __init__(self, d_model=256, d_ffn=1024, + dropout=0.1, activation="relu", + n_levels=4, n_heads=8, n_points=4): + super().__init__() + + # cross attention + self.cross_attn = MSDeformAttn(d_model, n_levels, n_heads, n_points) + self.dropout1 = nn.Dropout(dropout) + self.norm1 = nn.LayerNorm(d_model) + + # self attention + self.self_attn = nn.MultiheadAttention(d_model, n_heads, dropout=dropout) + self.dropout2 = nn.Dropout(dropout) + self.norm2 = nn.LayerNorm(d_model) + + # ffn + self.linear1 = nn.Linear(d_model, d_ffn) + self.activation = _get_activation_fn(activation) + self.dropout3 = nn.Dropout(dropout) + self.linear2 = nn.Linear(d_ffn, d_model) + self.dropout4 = nn.Dropout(dropout) + self.norm3 = nn.LayerNorm(d_model) + + @staticmethod + def with_pos_embed(tensor, pos): + return tensor if pos is None else tensor + pos + + def forward_ffn(self, tgt): + tgt2 = self.linear2(self.dropout3(self.activation(self.linear1(tgt)))) + tgt = tgt + self.dropout4(tgt2) + tgt = self.norm3(tgt) + return tgt + + def forward(self, tgt, query_pos, reference_points, src, src_spatial_shapes, level_start_index, src_padding_mask=None): + # self attention + q = k = self.with_pos_embed(tgt, query_pos) + tgt2 = self.self_attn(q.transpose(0, 1), k.transpose(0, 1), tgt.transpose(0, 1))[0].transpose(0, 1) + tgt = tgt + self.dropout2(tgt2) + tgt = self.norm2(tgt) + + # cross attention + tgt2 = self.cross_attn(self.with_pos_embed(tgt, query_pos), + reference_points, + src, src_spatial_shapes, level_start_index, src_padding_mask) + tgt = tgt + self.dropout1(tgt2) + tgt = self.norm1(tgt) + + # ffn + tgt = self.forward_ffn(tgt) + + return tgt + + +class DeformableTransformerDecoder(nn.Module): + def __init__(self, decoder_layer, num_layers, return_intermediate=False): + super().__init__() + self.layers = _get_clones(decoder_layer, num_layers) + self.num_layers = num_layers + self.return_intermediate = return_intermediate + # hack implementation for iterative bounding box refinement and two-stage Deformable DETR + self.bbox_embed = None + self.class_embed = None + + def forward(self, tgt, reference_points, src, src_spatial_shapes, src_level_start_index, src_valid_ratios, + query_pos=None, src_padding_mask=None): + output = tgt + + intermediate = [] + intermediate_reference_points = [] + for lid, layer in enumerate(self.layers): + if reference_points.shape[-1] == 4: + reference_points_input = reference_points[:, :, None] \ + * torch.cat([src_valid_ratios, src_valid_ratios], -1)[:, None] + else: + assert reference_points.shape[-1] == 2 + reference_points_input = reference_points[:, :, None] * src_valid_ratios[:, None] + output = layer(output, query_pos, reference_points_input, src, src_spatial_shapes, src_level_start_index, src_padding_mask) + + # hack implementation for iterative bounding box refinement + if self.bbox_embed is not None: + tmp = self.bbox_embed[lid](output) + if reference_points.shape[-1] == 4: + new_reference_points = tmp + inverse_sigmoid(reference_points) + new_reference_points = new_reference_points.sigmoid() + else: + assert reference_points.shape[-1] == 2 + new_reference_points = tmp + new_reference_points[..., :2] = tmp[..., :2] + inverse_sigmoid(reference_points) + new_reference_points = new_reference_points.sigmoid() + reference_points = new_reference_points.detach() + + if self.return_intermediate: + intermediate.append(output) + intermediate_reference_points.append(reference_points) + + if self.return_intermediate: + return torch.stack(intermediate), torch.stack(intermediate_reference_points) + + return output, reference_points + + +def _get_clones(module, N): + return nn.ModuleList([copy.deepcopy(module) for i in range(N)]) + + +def _get_activation_fn(activation): + """Return an activation function given a string""" + if activation == "relu": + return F.relu + if activation == "gelu": + return F.gelu + if activation == "glu": + return F.glu + raise RuntimeError(F"activation should be relu/gelu, not {activation}.") + + +def build_deforamble_transformer(args): + return DeformableTransformer( + d_model=args.hidden_dim, + nhead=args.nheads, + num_encoder_layers=args.enc_layers, + num_decoder_layers=args.dec_layers, + dim_feedforward=args.dim_feedforward, + dropout=args.dropout, + activation="relu", + return_intermediate_dec=True, + num_feature_levels=args.num_feature_levels, + dec_n_points=args.dec_n_points, + enc_n_points=args.enc_n_points, + two_stage=args.two_stage, + two_stage_num_proposals=args.num_queries, + assign_first_stage=args.assign_first_stage, + ) + + diff --git a/perception_models/apps/detection/DETA_pe/models/matcher.py b/perception_models/apps/detection/DETA_pe/models/matcher.py new file mode 100644 index 0000000000000000000000000000000000000000..63ef0294252e38d073d2c6d11e420e7cd306a7e2 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/models/matcher.py @@ -0,0 +1,102 @@ +# ------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------ +# Modified from DETR (https://github.com/facebookresearch/detr) +# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved +# ------------------------------------------------------------------------ + +""" +Modules to compute the matching cost and solve the corresponding LSAP. +""" +import torch +from scipy.optimize import linear_sum_assignment +from torch import nn + +from util.box_ops import box_cxcywh_to_xyxy, generalized_box_iou + + +class HungarianMatcher(nn.Module): + """This class computes an assignment between the targets and the predictions of the network + + For efficiency reasons, the targets don't include the no_object. Because of this, in general, + there are more predictions than targets. In this case, we do a 1-to-1 matching of the best predictions, + while the others are un-matched (and thus treated as non-objects). + """ + + def __init__(self, + cost_class: float = 1, + cost_bbox: float = 1, + cost_giou: float = 1): + """Creates the matcher + + Params: + cost_class: This is the relative weight of the classification error in the matching cost + cost_bbox: This is the relative weight of the L1 error of the bounding box coordinates in the matching cost + cost_giou: This is the relative weight of the giou loss of the bounding box in the matching cost + """ + super().__init__() + self.cost_class = cost_class + self.cost_bbox = cost_bbox + self.cost_giou = cost_giou + assert cost_class != 0 or cost_bbox != 0 or cost_giou != 0, "all costs cant be 0" + + def forward(self, outputs, targets): + """ Performs the matching + + Params: + outputs: This is a dict that contains at least these entries: + "pred_logits": Tensor of dim [batch_size, num_queries, num_classes] with the classification logits + "pred_boxes": Tensor of dim [batch_size, num_queries, 4] with the predicted box coordinates + + targets: This is a list of targets (len(targets) = batch_size), where each target is a dict containing: + "labels": Tensor of dim [num_target_boxes] (where num_target_boxes is the number of ground-truth + objects in the target) containing the class labels + "boxes": Tensor of dim [num_target_boxes, 4] containing the target box coordinates + + Returns: + A list of size batch_size, containing tuples of (index_i, index_j) where: + - index_i is the indices of the selected predictions (in order) + - index_j is the indices of the corresponding selected targets (in order) + For each batch element, it holds: + len(index_i) = len(index_j) = min(num_queries, num_target_boxes) + """ + with torch.no_grad(): + bs, num_queries = outputs["pred_logits"].shape[:2] + + # We flatten to compute the cost matrices in a batch + out_prob = outputs["pred_logits"].flatten(0, 1).sigmoid() + out_bbox = outputs["pred_boxes"].flatten(0, 1) # [batch_size * num_queries, 4] + + # Also concat the target labels and boxes + tgt_ids = torch.cat([v["labels"] for v in targets]) + tgt_bbox = torch.cat([v["boxes"] for v in targets]) + + # Compute the classification cost. + alpha = 0.25 + gamma = 2.0 + neg_cost_class = (1 - alpha) * (out_prob ** gamma) * (-(1 - out_prob + 1e-8).log()) + pos_cost_class = alpha * ((1 - out_prob) ** gamma) * (-(out_prob + 1e-8).log()) + cost_class = pos_cost_class[:, tgt_ids] - neg_cost_class[:, tgt_ids] + + # Compute the L1 cost between boxes + cost_bbox = torch.cdist(out_bbox, tgt_bbox, p=1) + + # Compute the giou cost betwen boxes + cost_giou = -generalized_box_iou(box_cxcywh_to_xyxy(out_bbox), + box_cxcywh_to_xyxy(tgt_bbox)) + + # Final cost matrix + C = self.cost_bbox * cost_bbox + self.cost_class * cost_class + self.cost_giou * cost_giou + C = C.view(bs, num_queries, -1).cpu() + + sizes = [len(v["boxes"]) for v in targets] + indices = [linear_sum_assignment(c[i]) for i, c in enumerate(C.split(sizes, -1))] + return [(torch.as_tensor(i, dtype=torch.int64), torch.as_tensor(j, dtype=torch.int64)) for i, j in indices] + + +def build_matcher(args): + return HungarianMatcher(cost_class=args.set_cost_class, + cost_bbox=args.set_cost_bbox, + cost_giou=args.set_cost_giou) diff --git a/perception_models/apps/detection/DETA_pe/models/ops/functions/__init__.py b/perception_models/apps/detection/DETA_pe/models/ops/functions/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..303161cd10022f2db15985b6da9b4d6571572656 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/models/ops/functions/__init__.py @@ -0,0 +1,9 @@ +# ------------------------------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------------------------------ +# Modified from https://github.com/chengdazhi/Deformable-Convolution-V2-PyTorch/tree/pytorch_1.0.0 +# ------------------------------------------------------------------------------------------------ + +from .ms_deform_attn_func import ms_deform_attn_core_pytorch, MSDeformAttnFunction diff --git a/perception_models/apps/detection/DETA_pe/models/ops/functions/ms_deform_attn_func.py b/perception_models/apps/detection/DETA_pe/models/ops/functions/ms_deform_attn_func.py new file mode 100644 index 0000000000000000000000000000000000000000..e6e80f4ab12d384d9d2a3629b7f0b0c0adb3b591 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/models/ops/functions/ms_deform_attn_func.py @@ -0,0 +1,106 @@ +# ------------------------------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------------------------------ +# Modified from https://github.com/chengdazhi/Deformable-Convolution-V2-PyTorch/tree/pytorch_1.0.0 +# ------------------------------------------------------------------------------------------------ + +from __future__ import absolute_import, division, print_function + +import MultiScaleDeformableAttention as MSDA + +import torch +import torch.nn.functional as F +from torch.autograd import Function +from torch.autograd.function import once_differentiable + + +class MSDeformAttnFunction(Function): + @staticmethod + def forward( + ctx, + value, + value_spatial_shapes, + value_level_start_index, + sampling_locations, + attention_weights, + im2col_step, + ): + ctx.im2col_step = im2col_step + output = MSDA.ms_deform_attn_forward( + value, + value_spatial_shapes, + value_level_start_index, + sampling_locations, + attention_weights, + ctx.im2col_step, + ) + ctx.save_for_backward( + value, + value_spatial_shapes, + value_level_start_index, + sampling_locations, + attention_weights, + ) + return output + + @staticmethod + @once_differentiable + def backward(ctx, grad_output): + ( + value, + value_spatial_shapes, + value_level_start_index, + sampling_locations, + attention_weights, + ) = ctx.saved_tensors + grad_value, grad_sampling_loc, grad_attn_weight = MSDA.ms_deform_attn_backward( + value, + value_spatial_shapes, + value_level_start_index, + sampling_locations, + attention_weights, + grad_output, + ctx.im2col_step, + ) + + return grad_value, None, None, grad_sampling_loc, grad_attn_weight, None + + +def ms_deform_attn_core_pytorch( + value, value_spatial_shapes, sampling_locations, attention_weights +): + # for debug and test only, + # need to use cuda version instead + N_, S_, M_, D_ = value.shape + _, Lq_, M_, L_, P_, _ = sampling_locations.shape + value_list = value.split([H_ * W_ for H_, W_ in value_spatial_shapes], dim=1) + sampling_grids = 2 * sampling_locations - 1 + sampling_value_list = [] + for lid_, (H_, W_) in enumerate(value_spatial_shapes): + # N_, H_*W_, M_, D_ -> N_, H_*W_, M_*D_ -> N_, M_*D_, H_*W_ -> N_*M_, D_, H_, W_ + value_l_ = ( + value_list[lid_].flatten(2).transpose(1, 2).reshape(N_ * M_, D_, H_, W_) + ) + # N_, Lq_, M_, P_, 2 -> N_, M_, Lq_, P_, 2 -> N_*M_, Lq_, P_, 2 + sampling_grid_l_ = sampling_grids[:, :, :, lid_].transpose(1, 2).flatten(0, 1) + # N_*M_, D_, Lq_, P_ + sampling_value_l_ = F.grid_sample( + value_l_, + sampling_grid_l_, + mode="bilinear", + padding_mode="zeros", + align_corners=False, + ) + sampling_value_list.append(sampling_value_l_) + # (N_, Lq_, M_, L_, P_) -> (N_, M_, Lq_, L_, P_) -> (N_, M_, 1, Lq_, L_*P_) + attention_weights = attention_weights.transpose(1, 2).reshape( + N_ * M_, 1, Lq_, L_ * P_ + ) + output = ( + (torch.stack(sampling_value_list, dim=-2).flatten(-2) * attention_weights) + .sum(-1) + .view(N_, M_ * D_, Lq_) + ) + return output.transpose(1, 2).contiguous() diff --git a/perception_models/apps/detection/DETA_pe/models/ops/make.sh b/perception_models/apps/detection/DETA_pe/models/ops/make.sh new file mode 100644 index 0000000000000000000000000000000000000000..106b685722bc6ed70a06bf04309e75e62f73a430 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/models/ops/make.sh @@ -0,0 +1,10 @@ +#!/usr/bin/env bash +# ------------------------------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------------------------------ +# Modified from https://github.com/chengdazhi/Deformable-Convolution-V2-PyTorch/tree/pytorch_1.0.0 +# ------------------------------------------------------------------------------------------------ + +python setup.py build install diff --git a/perception_models/apps/detection/DETA_pe/models/ops/modules/__init__.py b/perception_models/apps/detection/DETA_pe/models/ops/modules/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..f82cb1ad9d634a87b54ba6a71b58a230bcade5fe --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/models/ops/modules/__init__.py @@ -0,0 +1,9 @@ +# ------------------------------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------------------------------ +# Modified from https://github.com/chengdazhi/Deformable-Convolution-V2-PyTorch/tree/pytorch_1.0.0 +# ------------------------------------------------------------------------------------------------ + +from .ms_deform_attn import MSDeformAttn diff --git a/perception_models/apps/detection/DETA_pe/models/ops/modules/ms_deform_attn.py b/perception_models/apps/detection/DETA_pe/models/ops/modules/ms_deform_attn.py new file mode 100644 index 0000000000000000000000000000000000000000..905f7f19ce9a8fcf7f94c42d45c54411c58373c2 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/models/ops/modules/ms_deform_attn.py @@ -0,0 +1,161 @@ +# ------------------------------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------------------------------ +# Modified from https://github.com/chengdazhi/Deformable-Convolution-V2-PyTorch/tree/pytorch_1.0.0 +# ------------------------------------------------------------------------------------------------ + +from __future__ import absolute_import, division, print_function + +import math + +import warnings + +import torch +import torch.nn.functional as F +from torch import nn +from torch.nn.init import constant_, xavier_uniform_ + +from ..functions import ms_deform_attn_core_pytorch, MSDeformAttnFunction + + +def _is_power_of_2(n): + if (not isinstance(n, int)) or (n < 0): + raise ValueError( + "invalid input for _is_power_of_2: {} (type: {})".format(n, type(n)) + ) + return (n & (n - 1) == 0) and n != 0 + + +class MSDeformAttn(nn.Module): + def __init__(self, d_model=256, n_levels=4, n_heads=8, n_points=4): + """ + Multi-Scale Deformable Attention Module + :param d_model hidden dimension + :param n_levels number of feature levels + :param n_heads number of attention heads + :param n_points number of sampling points per attention head per feature level + """ + super().__init__() + if d_model % n_heads != 0: + raise ValueError( + "d_model must be divisible by n_heads, but got {} and {}".format( + d_model, n_heads + ) + ) + _d_per_head = d_model // n_heads + # you'd better set _d_per_head to a power of 2 which is more efficient in our CUDA implementation + if not _is_power_of_2(_d_per_head): + warnings.warn( + "You'd better set d_model in MSDeformAttn to make the dimension of each attention head a power of 2 " + "which is more efficient in our CUDA implementation." + ) + + self.im2col_step = 64 + + self.d_model = d_model + self.n_levels = n_levels + self.n_heads = n_heads + self.n_points = n_points + + self.sampling_offsets = nn.Linear(d_model, n_heads * n_levels * n_points * 2) + self.attention_weights = nn.Linear(d_model, n_heads * n_levels * n_points) + self.value_proj = nn.Linear(d_model, d_model) + self.output_proj = nn.Linear(d_model, d_model) + + self._reset_parameters() + + def _reset_parameters(self): + constant_(self.sampling_offsets.weight.data, 0.0) + thetas = torch.arange(self.n_heads, dtype=torch.float32) * ( + 2.0 * math.pi / self.n_heads + ) + grid_init = torch.stack([thetas.cos(), thetas.sin()], -1) + grid_init = ( + (grid_init / grid_init.abs().max(-1, keepdim=True)[0]) + .view(self.n_heads, 1, 1, 2) + .repeat(1, self.n_levels, self.n_points, 1) + ) + for i in range(self.n_points): + grid_init[:, :, i, :] *= i + 1 + with torch.no_grad(): + self.sampling_offsets.bias = nn.Parameter(grid_init.view(-1)) + constant_(self.attention_weights.weight.data, 0.0) + constant_(self.attention_weights.bias.data, 0.0) + xavier_uniform_(self.value_proj.weight.data) + constant_(self.value_proj.bias.data, 0.0) + xavier_uniform_(self.output_proj.weight.data) + constant_(self.output_proj.bias.data, 0.0) + + def forward( + self, + query, + reference_points, + input_flatten, + input_spatial_shapes, + input_level_start_index, + input_padding_mask=None, + ): + """ + :param query (N, Length_{query}, C) + :param reference_points (N, Length_{query}, n_levels, 2), range in [0, 1], top-left (0,0), bottom-right (1, 1), including padding area + or (N, Length_{query}, n_levels, 4), add additional (w, h) to form reference boxes + :param input_flatten (N, \sum_{l=0}^{L-1} H_l \cdot W_l, C) + :param input_spatial_shapes (n_levels, 2), [(H_0, W_0), (H_1, W_1), ..., (H_{L-1}, W_{L-1})] + :param input_level_start_index (n_levels, ), [0, H_0*W_0, H_0*W_0+H_1*W_1, H_0*W_0+H_1*W_1+H_2*W_2, ..., H_0*W_0+H_1*W_1+...+H_{L-1}*W_{L-1}] + :param input_padding_mask (N, \sum_{l=0}^{L-1} H_l \cdot W_l), True for padding elements, False for non-padding elements + + :return output (N, Length_{query}, C) + """ + N, Len_q, _ = query.shape + N, Len_in, _ = input_flatten.shape + assert (input_spatial_shapes[:, 0] * input_spatial_shapes[:, 1]).sum() == Len_in + + value = self.value_proj(input_flatten) + if input_padding_mask is not None: + value = value.masked_fill(input_padding_mask[..., None], float(0)) + value = value.view(N, Len_in, self.n_heads, self.d_model // self.n_heads) + sampling_offsets = self.sampling_offsets(query).view( + N, Len_q, self.n_heads, self.n_levels, self.n_points, 2 + ) + attention_weights = self.attention_weights(query).view( + N, Len_q, self.n_heads, self.n_levels * self.n_points + ) + attention_weights = F.softmax(attention_weights, -1).view( + N, Len_q, self.n_heads, self.n_levels, self.n_points + ) + # N, Len_q, n_heads, n_levels, n_points, 2 + if reference_points.shape[-1] == 2: + offset_normalizer = torch.stack( + [input_spatial_shapes[..., 1], input_spatial_shapes[..., 0]], -1 + ) + sampling_locations = ( + reference_points[:, :, None, :, None, :] + + sampling_offsets / offset_normalizer[None, None, None, :, None, :] + ) + elif reference_points.shape[-1] == 4: + sampling_locations = ( + reference_points[:, :, None, :, None, :2] + + sampling_offsets + / self.n_points + * reference_points[:, :, None, :, None, 2:] + * 0.5 + ) + else: + raise ValueError( + "Last dim of reference_points must be 2 or 4, but get {} instead.".format( + reference_points.shape[-1] + ) + ) + output = MSDeformAttnFunction.apply( + value, + input_spatial_shapes, + input_level_start_index, + sampling_locations, + attention_weights, + self.im2col_step, + ) + + output = self.output_proj(output) + return output diff --git a/perception_models/apps/detection/DETA_pe/models/ops/setup.py b/perception_models/apps/detection/DETA_pe/models/ops/setup.py new file mode 100644 index 0000000000000000000000000000000000000000..a0131bc21cf1b45b90fcf174e2c53e4c08e9c641 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/models/ops/setup.py @@ -0,0 +1,71 @@ +# ------------------------------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------------------------------ +# Modified from https://github.com/chengdazhi/Deformable-Convolution-V2-PyTorch/tree/pytorch_1.0.0 +# ------------------------------------------------------------------------------------------------ + +import os +import glob + +import torch + +from torch.utils.cpp_extension import CUDA_HOME +from torch.utils.cpp_extension import CppExtension +from torch.utils.cpp_extension import CUDAExtension + +from setuptools import find_packages +from setuptools import setup + +requirements = ["torch", "torchvision"] + +def get_extensions(): + this_dir = os.path.dirname(os.path.abspath(__file__)) + extensions_dir = os.path.join(this_dir, "src") + + main_file = glob.glob(os.path.join(extensions_dir, "*.cpp")) + source_cpu = glob.glob(os.path.join(extensions_dir, "cpu", "*.cpp")) + source_cuda = glob.glob(os.path.join(extensions_dir, "cuda", "*.cu")) + + sources = main_file + source_cpu + extension = CppExtension + extra_compile_args = {"cxx": []} + define_macros = [] + + if torch.cuda.is_available() and CUDA_HOME is not None: + extension = CUDAExtension + sources += source_cuda + define_macros += [("WITH_CUDA", None)] + extra_compile_args["nvcc"] = [ + "-DCUDA_HAS_FP16=1", + "-D__CUDA_NO_HALF_OPERATORS__", + "-D__CUDA_NO_HALF_CONVERSIONS__", + "-D__CUDA_NO_HALF2_OPERATORS__", + ] + else: + raise NotImplementedError('Cuda is not availabel') + + sources = [os.path.join(extensions_dir, s) for s in sources] + include_dirs = [extensions_dir] + ext_modules = [ + extension( + "MultiScaleDeformableAttention", + sources, + include_dirs=include_dirs, + define_macros=define_macros, + extra_compile_args=extra_compile_args, + ) + ] + return ext_modules + +setup( + name="MultiScaleDeformableAttention", + version="1.0", + author="Weijie Su", + url="https://github.com/fundamentalvision/Deformable-DETR", + description="PyTorch Wrapper for CUDA Functions of Multi-Scale Deformable Attention", + packages=find_packages(exclude=("configs", "tests",)), + ext_modules=get_extensions(), + cmdclass={"build_ext": torch.utils.cpp_extension.BuildExtension}, +) diff --git a/perception_models/apps/detection/DETA_pe/models/ops/src/cpu/ms_deform_attn_cpu.cpp b/perception_models/apps/detection/DETA_pe/models/ops/src/cpu/ms_deform_attn_cpu.cpp new file mode 100644 index 0000000000000000000000000000000000000000..e1bf854de1f3860d20b6fef5c1a17817c268e70a --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/models/ops/src/cpu/ms_deform_attn_cpu.cpp @@ -0,0 +1,41 @@ +/*! +************************************************************************************************** +* Deformable DETR +* Copyright (c) 2020 SenseTime. All Rights Reserved. +* Licensed under the Apache License, Version 2.0 [see LICENSE for details] +************************************************************************************************** +* Modified from https://github.com/chengdazhi/Deformable-Convolution-V2-PyTorch/tree/pytorch_1.0.0 +************************************************************************************************** +*/ + +#include + +#include +#include + + +at::Tensor +ms_deform_attn_cpu_forward( + const at::Tensor &value, + const at::Tensor &spatial_shapes, + const at::Tensor &level_start_index, + const at::Tensor &sampling_loc, + const at::Tensor &attn_weight, + const int im2col_step) +{ + AT_ERROR("Not implement on cpu"); +} + +std::vector +ms_deform_attn_cpu_backward( + const at::Tensor &value, + const at::Tensor &spatial_shapes, + const at::Tensor &level_start_index, + const at::Tensor &sampling_loc, + const at::Tensor &attn_weight, + const at::Tensor &grad_output, + const int im2col_step) +{ + AT_ERROR("Not implement on cpu"); +} + diff --git a/perception_models/apps/detection/DETA_pe/models/ops/src/cpu/ms_deform_attn_cpu.h b/perception_models/apps/detection/DETA_pe/models/ops/src/cpu/ms_deform_attn_cpu.h new file mode 100644 index 0000000000000000000000000000000000000000..81b7b58a3d9502bbb684dc84687a526dedf94cae --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/models/ops/src/cpu/ms_deform_attn_cpu.h @@ -0,0 +1,33 @@ +/*! +************************************************************************************************** +* Deformable DETR +* Copyright (c) 2020 SenseTime. All Rights Reserved. +* Licensed under the Apache License, Version 2.0 [see LICENSE for details] +************************************************************************************************** +* Modified from https://github.com/chengdazhi/Deformable-Convolution-V2-PyTorch/tree/pytorch_1.0.0 +************************************************************************************************** +*/ + +#pragma once +#include + +at::Tensor +ms_deform_attn_cpu_forward( + const at::Tensor &value, + const at::Tensor &spatial_shapes, + const at::Tensor &level_start_index, + const at::Tensor &sampling_loc, + const at::Tensor &attn_weight, + const int im2col_step); + +std::vector +ms_deform_attn_cpu_backward( + const at::Tensor &value, + const at::Tensor &spatial_shapes, + const at::Tensor &level_start_index, + const at::Tensor &sampling_loc, + const at::Tensor &attn_weight, + const at::Tensor &grad_output, + const int im2col_step); + + diff --git a/perception_models/apps/detection/DETA_pe/models/ops/src/cuda/ms_deform_attn_cuda.cu b/perception_models/apps/detection/DETA_pe/models/ops/src/cuda/ms_deform_attn_cuda.cu new file mode 100644 index 0000000000000000000000000000000000000000..d6d583647cce987196d5ad1968a8a365a379e774 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/models/ops/src/cuda/ms_deform_attn_cuda.cu @@ -0,0 +1,153 @@ +/*! +************************************************************************************************** +* Deformable DETR +* Copyright (c) 2020 SenseTime. All Rights Reserved. +* Licensed under the Apache License, Version 2.0 [see LICENSE for details] +************************************************************************************************** +* Modified from https://github.com/chengdazhi/Deformable-Convolution-V2-PyTorch/tree/pytorch_1.0.0 +************************************************************************************************** +*/ + +#include +#include "cuda/ms_deform_im2col_cuda.cuh" + +#include +#include +#include +#include + + +at::Tensor ms_deform_attn_cuda_forward( + const at::Tensor &value, + const at::Tensor &spatial_shapes, + const at::Tensor &level_start_index, + const at::Tensor &sampling_loc, + const at::Tensor &attn_weight, + const int im2col_step) +{ + AT_ASSERTM(value.is_contiguous(), "value tensor has to be contiguous"); + AT_ASSERTM(spatial_shapes.is_contiguous(), "spatial_shapes tensor has to be contiguous"); + AT_ASSERTM(level_start_index.is_contiguous(), "level_start_index tensor has to be contiguous"); + AT_ASSERTM(sampling_loc.is_contiguous(), "sampling_loc tensor has to be contiguous"); + AT_ASSERTM(attn_weight.is_contiguous(), "attn_weight tensor has to be contiguous"); + + AT_ASSERTM(value.type().is_cuda(), "value must be a CUDA tensor"); + AT_ASSERTM(spatial_shapes.type().is_cuda(), "spatial_shapes must be a CUDA tensor"); + AT_ASSERTM(level_start_index.type().is_cuda(), "level_start_index must be a CUDA tensor"); + AT_ASSERTM(sampling_loc.type().is_cuda(), "sampling_loc must be a CUDA tensor"); + AT_ASSERTM(attn_weight.type().is_cuda(), "attn_weight must be a CUDA tensor"); + + const int batch = value.size(0); + const int spatial_size = value.size(1); + const int num_heads = value.size(2); + const int channels = value.size(3); + + const int num_levels = spatial_shapes.size(0); + + const int num_query = sampling_loc.size(1); + const int num_point = sampling_loc.size(4); + + const int im2col_step_ = std::min(batch, im2col_step); + + AT_ASSERTM(batch % im2col_step_ == 0, "batch(%d) must divide im2col_step(%d)", batch, im2col_step_); + + auto output = at::zeros({batch, num_query, num_heads, channels}, value.options()); + + const int batch_n = im2col_step_; + auto output_n = output.view({batch/im2col_step_, batch_n, num_query, num_heads, channels}); + auto per_value_size = spatial_size * num_heads * channels; + auto per_sample_loc_size = num_query * num_heads * num_levels * num_point * 2; + auto per_attn_weight_size = num_query * num_heads * num_levels * num_point; + for (int n = 0; n < batch/im2col_step_; ++n) + { + auto columns = output_n.select(0, n); + AT_DISPATCH_FLOATING_TYPES(value.type(), "ms_deform_attn_forward_cuda", ([&] { + ms_deformable_im2col_cuda(at::cuda::getCurrentCUDAStream(), + value.data() + n * im2col_step_ * per_value_size, + spatial_shapes.data(), + level_start_index.data(), + sampling_loc.data() + n * im2col_step_ * per_sample_loc_size, + attn_weight.data() + n * im2col_step_ * per_attn_weight_size, + batch_n, spatial_size, num_heads, channels, num_levels, num_query, num_point, + columns.data()); + + })); + } + + output = output.view({batch, num_query, num_heads*channels}); + + return output; +} + + +std::vector ms_deform_attn_cuda_backward( + const at::Tensor &value, + const at::Tensor &spatial_shapes, + const at::Tensor &level_start_index, + const at::Tensor &sampling_loc, + const at::Tensor &attn_weight, + const at::Tensor &grad_output, + const int im2col_step) +{ + + AT_ASSERTM(value.is_contiguous(), "value tensor has to be contiguous"); + AT_ASSERTM(spatial_shapes.is_contiguous(), "spatial_shapes tensor has to be contiguous"); + AT_ASSERTM(level_start_index.is_contiguous(), "level_start_index tensor has to be contiguous"); + AT_ASSERTM(sampling_loc.is_contiguous(), "sampling_loc tensor has to be contiguous"); + AT_ASSERTM(attn_weight.is_contiguous(), "attn_weight tensor has to be contiguous"); + AT_ASSERTM(grad_output.is_contiguous(), "grad_output tensor has to be contiguous"); + + AT_ASSERTM(value.type().is_cuda(), "value must be a CUDA tensor"); + AT_ASSERTM(spatial_shapes.type().is_cuda(), "spatial_shapes must be a CUDA tensor"); + AT_ASSERTM(level_start_index.type().is_cuda(), "level_start_index must be a CUDA tensor"); + AT_ASSERTM(sampling_loc.type().is_cuda(), "sampling_loc must be a CUDA tensor"); + AT_ASSERTM(attn_weight.type().is_cuda(), "attn_weight must be a CUDA tensor"); + AT_ASSERTM(grad_output.type().is_cuda(), "grad_output must be a CUDA tensor"); + + const int batch = value.size(0); + const int spatial_size = value.size(1); + const int num_heads = value.size(2); + const int channels = value.size(3); + + const int num_levels = spatial_shapes.size(0); + + const int num_query = sampling_loc.size(1); + const int num_point = sampling_loc.size(4); + + const int im2col_step_ = std::min(batch, im2col_step); + + AT_ASSERTM(batch % im2col_step_ == 0, "batch(%d) must divide im2col_step(%d)", batch, im2col_step_); + + auto grad_value = at::zeros_like(value); + auto grad_sampling_loc = at::zeros_like(sampling_loc); + auto grad_attn_weight = at::zeros_like(attn_weight); + + const int batch_n = im2col_step_; + auto per_value_size = spatial_size * num_heads * channels; + auto per_sample_loc_size = num_query * num_heads * num_levels * num_point * 2; + auto per_attn_weight_size = num_query * num_heads * num_levels * num_point; + auto grad_output_n = grad_output.view({batch/im2col_step_, batch_n, num_query, num_heads, channels}); + + for (int n = 0; n < batch/im2col_step_; ++n) + { + auto grad_output_g = grad_output_n.select(0, n); + AT_DISPATCH_FLOATING_TYPES(value.type(), "ms_deform_attn_backward_cuda", ([&] { + ms_deformable_col2im_cuda(at::cuda::getCurrentCUDAStream(), + grad_output_g.data(), + value.data() + n * im2col_step_ * per_value_size, + spatial_shapes.data(), + level_start_index.data(), + sampling_loc.data() + n * im2col_step_ * per_sample_loc_size, + attn_weight.data() + n * im2col_step_ * per_attn_weight_size, + batch_n, spatial_size, num_heads, channels, num_levels, num_query, num_point, + grad_value.data() + n * im2col_step_ * per_value_size, + grad_sampling_loc.data() + n * im2col_step_ * per_sample_loc_size, + grad_attn_weight.data() + n * im2col_step_ * per_attn_weight_size); + + })); + } + + return { + grad_value, grad_sampling_loc, grad_attn_weight + }; +} \ No newline at end of file diff --git a/perception_models/apps/detection/DETA_pe/models/ops/src/cuda/ms_deform_attn_cuda.h b/perception_models/apps/detection/DETA_pe/models/ops/src/cuda/ms_deform_attn_cuda.h new file mode 100644 index 0000000000000000000000000000000000000000..c7ae53f99c820ce6193b608ad344550348a0b42c --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/models/ops/src/cuda/ms_deform_attn_cuda.h @@ -0,0 +1,30 @@ +/*! +************************************************************************************************** +* Deformable DETR +* Copyright (c) 2020 SenseTime. All Rights Reserved. +* Licensed under the Apache License, Version 2.0 [see LICENSE for details] +************************************************************************************************** +* Modified from https://github.com/chengdazhi/Deformable-Convolution-V2-PyTorch/tree/pytorch_1.0.0 +************************************************************************************************** +*/ + +#pragma once +#include + +at::Tensor ms_deform_attn_cuda_forward( + const at::Tensor &value, + const at::Tensor &spatial_shapes, + const at::Tensor &level_start_index, + const at::Tensor &sampling_loc, + const at::Tensor &attn_weight, + const int im2col_step); + +std::vector ms_deform_attn_cuda_backward( + const at::Tensor &value, + const at::Tensor &spatial_shapes, + const at::Tensor &level_start_index, + const at::Tensor &sampling_loc, + const at::Tensor &attn_weight, + const at::Tensor &grad_output, + const int im2col_step); + diff --git a/perception_models/apps/detection/DETA_pe/models/ops/src/cuda/ms_deform_im2col_cuda.cuh b/perception_models/apps/detection/DETA_pe/models/ops/src/cuda/ms_deform_im2col_cuda.cuh new file mode 100644 index 0000000000000000000000000000000000000000..6bc2acb7aea0eab2e9e91e769a16861e1652c284 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/models/ops/src/cuda/ms_deform_im2col_cuda.cuh @@ -0,0 +1,1327 @@ +/*! +************************************************************************** +* Deformable DETR +* Copyright (c) 2020 SenseTime. All Rights Reserved. +* Licensed under the Apache License, Version 2.0 [see LICENSE for details] +************************************************************************** +* Modified from DCN (https://github.com/msracver/Deformable-ConvNets) +* Copyright (c) 2018 Microsoft +************************************************************************** +*/ + +#include +#include +#include + +#include +#include + +#include + +#define CUDA_KERNEL_LOOP(i, n) \ + for (int i = blockIdx.x * blockDim.x + threadIdx.x; \ + i < (n); \ + i += blockDim.x * gridDim.x) + +const int CUDA_NUM_THREADS = 1024; +inline int GET_BLOCKS(const int N, const int num_threads) +{ + return (N + num_threads - 1) / num_threads; +} + + +template +__device__ scalar_t ms_deform_attn_im2col_bilinear(const scalar_t* &bottom_data, + const int &height, const int &width, const int &nheads, const int &channels, + const scalar_t &h, const scalar_t &w, const int &m, const int &c) +{ + const int h_low = floor(h); + const int w_low = floor(w); + const int h_high = h_low + 1; + const int w_high = w_low + 1; + + const scalar_t lh = h - h_low; + const scalar_t lw = w - w_low; + const scalar_t hh = 1 - lh, hw = 1 - lw; + + const int w_stride = nheads * channels; + const int h_stride = width * w_stride; + const int h_low_ptr_offset = h_low * h_stride; + const int h_high_ptr_offset = h_low_ptr_offset + h_stride; + const int w_low_ptr_offset = w_low * w_stride; + const int w_high_ptr_offset = w_low_ptr_offset + w_stride; + const int base_ptr = m * channels + c; + + scalar_t v1 = 0; + if (h_low >= 0 && w_low >= 0) + { + const int ptr1 = h_low_ptr_offset + w_low_ptr_offset + base_ptr; + v1 = bottom_data[ptr1]; + } + scalar_t v2 = 0; + if (h_low >= 0 && w_high <= width - 1) + { + const int ptr2 = h_low_ptr_offset + w_high_ptr_offset + base_ptr; + v2 = bottom_data[ptr2]; + } + scalar_t v3 = 0; + if (h_high <= height - 1 && w_low >= 0) + { + const int ptr3 = h_high_ptr_offset + w_low_ptr_offset + base_ptr; + v3 = bottom_data[ptr3]; + } + scalar_t v4 = 0; + if (h_high <= height - 1 && w_high <= width - 1) + { + const int ptr4 = h_high_ptr_offset + w_high_ptr_offset + base_ptr; + v4 = bottom_data[ptr4]; + } + + const scalar_t w1 = hh * hw, w2 = hh * lw, w3 = lh * hw, w4 = lh * lw; + + const scalar_t val = (w1 * v1 + w2 * v2 + w3 * v3 + w4 * v4); + return val; +} + + +template +__device__ void ms_deform_attn_col2im_bilinear(const scalar_t* &bottom_data, + const int &height, const int &width, const int &nheads, const int &channels, + const scalar_t &h, const scalar_t &w, const int &m, const int &c, + const scalar_t &top_grad, + const scalar_t &attn_weight, + scalar_t* &grad_value, + scalar_t* grad_sampling_loc, + scalar_t* grad_attn_weight) +{ + const int h_low = floor(h); + const int w_low = floor(w); + const int h_high = h_low + 1; + const int w_high = w_low + 1; + + const scalar_t lh = h - h_low; + const scalar_t lw = w - w_low; + const scalar_t hh = 1 - lh, hw = 1 - lw; + + const int w_stride = nheads * channels; + const int h_stride = width * w_stride; + const int h_low_ptr_offset = h_low * h_stride; + const int h_high_ptr_offset = h_low_ptr_offset + h_stride; + const int w_low_ptr_offset = w_low * w_stride; + const int w_high_ptr_offset = w_low_ptr_offset + w_stride; + const int base_ptr = m * channels + c; + + const scalar_t w1 = hh * hw, w2 = hh * lw, w3 = lh * hw, w4 = lh * lw; + const scalar_t top_grad_value = top_grad * attn_weight; + scalar_t grad_h_weight = 0, grad_w_weight = 0; + + scalar_t v1 = 0; + if (h_low >= 0 && w_low >= 0) + { + const int ptr1 = h_low_ptr_offset + w_low_ptr_offset + base_ptr; + v1 = bottom_data[ptr1]; + grad_h_weight -= hw * v1; + grad_w_weight -= hh * v1; + atomicAdd(grad_value+ptr1, w1*top_grad_value); + } + scalar_t v2 = 0; + if (h_low >= 0 && w_high <= width - 1) + { + const int ptr2 = h_low_ptr_offset + w_high_ptr_offset + base_ptr; + v2 = bottom_data[ptr2]; + grad_h_weight -= lw * v2; + grad_w_weight += hh * v2; + atomicAdd(grad_value+ptr2, w2*top_grad_value); + } + scalar_t v3 = 0; + if (h_high <= height - 1 && w_low >= 0) + { + const int ptr3 = h_high_ptr_offset + w_low_ptr_offset + base_ptr; + v3 = bottom_data[ptr3]; + grad_h_weight += hw * v3; + grad_w_weight -= lh * v3; + atomicAdd(grad_value+ptr3, w3*top_grad_value); + } + scalar_t v4 = 0; + if (h_high <= height - 1 && w_high <= width - 1) + { + const int ptr4 = h_high_ptr_offset + w_high_ptr_offset + base_ptr; + v4 = bottom_data[ptr4]; + grad_h_weight += lw * v4; + grad_w_weight += lh * v4; + atomicAdd(grad_value+ptr4, w4*top_grad_value); + } + + const scalar_t val = (w1 * v1 + w2 * v2 + w3 * v3 + w4 * v4); + *grad_attn_weight = top_grad * val; + *grad_sampling_loc = width * grad_w_weight * top_grad_value; + *(grad_sampling_loc + 1) = height * grad_h_weight * top_grad_value; +} + + +template +__device__ void ms_deform_attn_col2im_bilinear_gm(const scalar_t* &bottom_data, + const int &height, const int &width, const int &nheads, const int &channels, + const scalar_t &h, const scalar_t &w, const int &m, const int &c, + const scalar_t &top_grad, + const scalar_t &attn_weight, + scalar_t* &grad_value, + scalar_t* grad_sampling_loc, + scalar_t* grad_attn_weight) +{ + const int h_low = floor(h); + const int w_low = floor(w); + const int h_high = h_low + 1; + const int w_high = w_low + 1; + + const scalar_t lh = h - h_low; + const scalar_t lw = w - w_low; + const scalar_t hh = 1 - lh, hw = 1 - lw; + + const int w_stride = nheads * channels; + const int h_stride = width * w_stride; + const int h_low_ptr_offset = h_low * h_stride; + const int h_high_ptr_offset = h_low_ptr_offset + h_stride; + const int w_low_ptr_offset = w_low * w_stride; + const int w_high_ptr_offset = w_low_ptr_offset + w_stride; + const int base_ptr = m * channels + c; + + const scalar_t w1 = hh * hw, w2 = hh * lw, w3 = lh * hw, w4 = lh * lw; + const scalar_t top_grad_value = top_grad * attn_weight; + scalar_t grad_h_weight = 0, grad_w_weight = 0; + + scalar_t v1 = 0; + if (h_low >= 0 && w_low >= 0) + { + const int ptr1 = h_low_ptr_offset + w_low_ptr_offset + base_ptr; + v1 = bottom_data[ptr1]; + grad_h_weight -= hw * v1; + grad_w_weight -= hh * v1; + atomicAdd(grad_value+ptr1, w1*top_grad_value); + } + scalar_t v2 = 0; + if (h_low >= 0 && w_high <= width - 1) + { + const int ptr2 = h_low_ptr_offset + w_high_ptr_offset + base_ptr; + v2 = bottom_data[ptr2]; + grad_h_weight -= lw * v2; + grad_w_weight += hh * v2; + atomicAdd(grad_value+ptr2, w2*top_grad_value); + } + scalar_t v3 = 0; + if (h_high <= height - 1 && w_low >= 0) + { + const int ptr3 = h_high_ptr_offset + w_low_ptr_offset + base_ptr; + v3 = bottom_data[ptr3]; + grad_h_weight += hw * v3; + grad_w_weight -= lh * v3; + atomicAdd(grad_value+ptr3, w3*top_grad_value); + } + scalar_t v4 = 0; + if (h_high <= height - 1 && w_high <= width - 1) + { + const int ptr4 = h_high_ptr_offset + w_high_ptr_offset + base_ptr; + v4 = bottom_data[ptr4]; + grad_h_weight += lw * v4; + grad_w_weight += lh * v4; + atomicAdd(grad_value+ptr4, w4*top_grad_value); + } + + const scalar_t val = (w1 * v1 + w2 * v2 + w3 * v3 + w4 * v4); + atomicAdd(grad_attn_weight, top_grad * val); + atomicAdd(grad_sampling_loc, width * grad_w_weight * top_grad_value); + atomicAdd(grad_sampling_loc + 1, height * grad_h_weight * top_grad_value); +} + + +template +__global__ void ms_deformable_im2col_gpu_kernel(const int n, + const scalar_t *data_value, + const int64_t *data_spatial_shapes, + const int64_t *data_level_start_index, + const scalar_t *data_sampling_loc, + const scalar_t *data_attn_weight, + const int batch_size, + const int spatial_size, + const int num_heads, + const int channels, + const int num_levels, + const int num_query, + const int num_point, + scalar_t *data_col) +{ + CUDA_KERNEL_LOOP(index, n) + { + int _temp = index; + const int c_col = _temp % channels; + _temp /= channels; + const int sampling_index = _temp; + const int m_col = _temp % num_heads; + _temp /= num_heads; + const int q_col = _temp % num_query; + _temp /= num_query; + const int b_col = _temp; + + scalar_t *data_col_ptr = data_col + index; + int data_weight_ptr = sampling_index * num_levels * num_point; + int data_loc_w_ptr = data_weight_ptr << 1; + const int qid_stride = num_heads * channels; + const int data_value_ptr_init_offset = b_col * spatial_size * qid_stride; + scalar_t col = 0; + + for (int l_col=0; l_col < num_levels; ++l_col) + { + const int level_start_id = data_level_start_index[l_col]; + const int spatial_h_ptr = l_col << 1; + const int spatial_h = data_spatial_shapes[spatial_h_ptr]; + const int spatial_w = data_spatial_shapes[spatial_h_ptr + 1]; + const scalar_t *data_value_ptr = data_value + (data_value_ptr_init_offset + level_start_id * qid_stride); + for (int p_col=0; p_col < num_point; ++p_col) + { + const scalar_t loc_w = data_sampling_loc[data_loc_w_ptr]; + const scalar_t loc_h = data_sampling_loc[data_loc_w_ptr + 1]; + const scalar_t weight = data_attn_weight[data_weight_ptr]; + + const scalar_t h_im = loc_h * spatial_h - 0.5; + const scalar_t w_im = loc_w * spatial_w - 0.5; + + if (h_im > -1 && w_im > -1 && h_im < spatial_h && w_im < spatial_w) + { + col += ms_deform_attn_im2col_bilinear(data_value_ptr, spatial_h, spatial_w, num_heads, channels, h_im, w_im, m_col, c_col) * weight; + } + + data_weight_ptr += 1; + data_loc_w_ptr += 2; + } + } + *data_col_ptr = col; + } +} + +template +__global__ void ms_deformable_col2im_gpu_kernel_shm_blocksize_aware_reduce_v1(const int n, + const scalar_t *grad_col, + const scalar_t *data_value, + const int64_t *data_spatial_shapes, + const int64_t *data_level_start_index, + const scalar_t *data_sampling_loc, + const scalar_t *data_attn_weight, + const int batch_size, + const int spatial_size, + const int num_heads, + const int channels, + const int num_levels, + const int num_query, + const int num_point, + scalar_t *grad_value, + scalar_t *grad_sampling_loc, + scalar_t *grad_attn_weight) +{ + CUDA_KERNEL_LOOP(index, n) + { + __shared__ scalar_t cache_grad_sampling_loc[blockSize * 2]; + __shared__ scalar_t cache_grad_attn_weight[blockSize]; + unsigned int tid = threadIdx.x; + int _temp = index; + const int c_col = _temp % channels; + _temp /= channels; + const int sampling_index = _temp; + const int m_col = _temp % num_heads; + _temp /= num_heads; + const int q_col = _temp % num_query; + _temp /= num_query; + const int b_col = _temp; + + const scalar_t top_grad = grad_col[index]; + + int data_weight_ptr = sampling_index * num_levels * num_point; + int data_loc_w_ptr = data_weight_ptr << 1; + const int grad_sampling_ptr = data_weight_ptr; + grad_sampling_loc += grad_sampling_ptr << 1; + grad_attn_weight += grad_sampling_ptr; + const int grad_weight_stride = 1; + const int grad_loc_stride = 2; + const int qid_stride = num_heads * channels; + const int data_value_ptr_init_offset = b_col * spatial_size * qid_stride; + + for (int l_col=0; l_col < num_levels; ++l_col) + { + const int level_start_id = data_level_start_index[l_col]; + const int spatial_h_ptr = l_col << 1; + const int spatial_h = data_spatial_shapes[spatial_h_ptr]; + const int spatial_w = data_spatial_shapes[spatial_h_ptr + 1]; + const int value_ptr_offset = data_value_ptr_init_offset + level_start_id * qid_stride; + const scalar_t *data_value_ptr = data_value + value_ptr_offset; + scalar_t *grad_value_ptr = grad_value + value_ptr_offset; + + for (int p_col=0; p_col < num_point; ++p_col) + { + const scalar_t loc_w = data_sampling_loc[data_loc_w_ptr]; + const scalar_t loc_h = data_sampling_loc[data_loc_w_ptr + 1]; + const scalar_t weight = data_attn_weight[data_weight_ptr]; + + const scalar_t h_im = loc_h * spatial_h - 0.5; + const scalar_t w_im = loc_w * spatial_w - 0.5; + *(cache_grad_sampling_loc+(threadIdx.x << 1)) = 0; + *(cache_grad_sampling_loc+((threadIdx.x << 1) + 1)) = 0; + *(cache_grad_attn_weight+threadIdx.x)=0; + if (h_im > -1 && w_im > -1 && h_im < spatial_h && w_im < spatial_w) + { + ms_deform_attn_col2im_bilinear( + data_value_ptr, spatial_h, spatial_w, num_heads, channels, h_im, w_im, m_col, c_col, + top_grad, weight, grad_value_ptr, + cache_grad_sampling_loc+(threadIdx.x << 1), cache_grad_attn_weight+threadIdx.x); + } + + __syncthreads(); + if (tid == 0) + { + scalar_t _grad_w=cache_grad_sampling_loc[0], _grad_h=cache_grad_sampling_loc[1], _grad_a=cache_grad_attn_weight[0]; + int sid=2; + for (unsigned int tid = 1; tid < blockSize; ++tid) + { + _grad_w += cache_grad_sampling_loc[sid]; + _grad_h += cache_grad_sampling_loc[sid + 1]; + _grad_a += cache_grad_attn_weight[tid]; + sid += 2; + } + + + *grad_sampling_loc = _grad_w; + *(grad_sampling_loc + 1) = _grad_h; + *grad_attn_weight = _grad_a; + } + __syncthreads(); + + data_weight_ptr += 1; + data_loc_w_ptr += 2; + grad_attn_weight += grad_weight_stride; + grad_sampling_loc += grad_loc_stride; + } + } + } +} + + +template +__global__ void ms_deformable_col2im_gpu_kernel_shm_blocksize_aware_reduce_v2(const int n, + const scalar_t *grad_col, + const scalar_t *data_value, + const int64_t *data_spatial_shapes, + const int64_t *data_level_start_index, + const scalar_t *data_sampling_loc, + const scalar_t *data_attn_weight, + const int batch_size, + const int spatial_size, + const int num_heads, + const int channels, + const int num_levels, + const int num_query, + const int num_point, + scalar_t *grad_value, + scalar_t *grad_sampling_loc, + scalar_t *grad_attn_weight) +{ + CUDA_KERNEL_LOOP(index, n) + { + __shared__ scalar_t cache_grad_sampling_loc[blockSize * 2]; + __shared__ scalar_t cache_grad_attn_weight[blockSize]; + unsigned int tid = threadIdx.x; + int _temp = index; + const int c_col = _temp % channels; + _temp /= channels; + const int sampling_index = _temp; + const int m_col = _temp % num_heads; + _temp /= num_heads; + const int q_col = _temp % num_query; + _temp /= num_query; + const int b_col = _temp; + + const scalar_t top_grad = grad_col[index]; + + int data_weight_ptr = sampling_index * num_levels * num_point; + int data_loc_w_ptr = data_weight_ptr << 1; + const int grad_sampling_ptr = data_weight_ptr; + grad_sampling_loc += grad_sampling_ptr << 1; + grad_attn_weight += grad_sampling_ptr; + const int grad_weight_stride = 1; + const int grad_loc_stride = 2; + const int qid_stride = num_heads * channels; + const int data_value_ptr_init_offset = b_col * spatial_size * qid_stride; + + for (int l_col=0; l_col < num_levels; ++l_col) + { + const int level_start_id = data_level_start_index[l_col]; + const int spatial_h_ptr = l_col << 1; + const int spatial_h = data_spatial_shapes[spatial_h_ptr]; + const int spatial_w = data_spatial_shapes[spatial_h_ptr + 1]; + const int value_ptr_offset = data_value_ptr_init_offset + level_start_id * qid_stride; + const scalar_t *data_value_ptr = data_value + value_ptr_offset; + scalar_t *grad_value_ptr = grad_value + value_ptr_offset; + + for (int p_col=0; p_col < num_point; ++p_col) + { + const scalar_t loc_w = data_sampling_loc[data_loc_w_ptr]; + const scalar_t loc_h = data_sampling_loc[data_loc_w_ptr + 1]; + const scalar_t weight = data_attn_weight[data_weight_ptr]; + + const scalar_t h_im = loc_h * spatial_h - 0.5; + const scalar_t w_im = loc_w * spatial_w - 0.5; + *(cache_grad_sampling_loc+(threadIdx.x << 1)) = 0; + *(cache_grad_sampling_loc+((threadIdx.x << 1) + 1)) = 0; + *(cache_grad_attn_weight+threadIdx.x)=0; + if (h_im > -1 && w_im > -1 && h_im < spatial_h && w_im < spatial_w) + { + ms_deform_attn_col2im_bilinear( + data_value_ptr, spatial_h, spatial_w, num_heads, channels, h_im, w_im, m_col, c_col, + top_grad, weight, grad_value_ptr, + cache_grad_sampling_loc+(threadIdx.x << 1), cache_grad_attn_weight+threadIdx.x); + } + + __syncthreads(); + + for (unsigned int s=blockSize/2; s>0; s>>=1) + { + if (tid < s) { + const unsigned int xid1 = tid << 1; + const unsigned int xid2 = (tid + s) << 1; + cache_grad_attn_weight[tid] += cache_grad_attn_weight[tid + s]; + cache_grad_sampling_loc[xid1] += cache_grad_sampling_loc[xid2]; + cache_grad_sampling_loc[xid1 + 1] += cache_grad_sampling_loc[xid2 + 1]; + } + __syncthreads(); + } + + if (tid == 0) + { + *grad_sampling_loc = cache_grad_sampling_loc[0]; + *(grad_sampling_loc + 1) = cache_grad_sampling_loc[1]; + *grad_attn_weight = cache_grad_attn_weight[0]; + } + __syncthreads(); + + data_weight_ptr += 1; + data_loc_w_ptr += 2; + grad_attn_weight += grad_weight_stride; + grad_sampling_loc += grad_loc_stride; + } + } + } +} + + +template +__global__ void ms_deformable_col2im_gpu_kernel_shm_reduce_v1(const int n, + const scalar_t *grad_col, + const scalar_t *data_value, + const int64_t *data_spatial_shapes, + const int64_t *data_level_start_index, + const scalar_t *data_sampling_loc, + const scalar_t *data_attn_weight, + const int batch_size, + const int spatial_size, + const int num_heads, + const int channels, + const int num_levels, + const int num_query, + const int num_point, + scalar_t *grad_value, + scalar_t *grad_sampling_loc, + scalar_t *grad_attn_weight) +{ + CUDA_KERNEL_LOOP(index, n) + { + extern __shared__ int _s[]; + scalar_t* cache_grad_sampling_loc = (scalar_t*)_s; + scalar_t* cache_grad_attn_weight = cache_grad_sampling_loc + 2 * blockDim.x; + unsigned int tid = threadIdx.x; + int _temp = index; + const int c_col = _temp % channels; + _temp /= channels; + const int sampling_index = _temp; + const int m_col = _temp % num_heads; + _temp /= num_heads; + const int q_col = _temp % num_query; + _temp /= num_query; + const int b_col = _temp; + + const scalar_t top_grad = grad_col[index]; + + int data_weight_ptr = sampling_index * num_levels * num_point; + int data_loc_w_ptr = data_weight_ptr << 1; + const int grad_sampling_ptr = data_weight_ptr; + grad_sampling_loc += grad_sampling_ptr << 1; + grad_attn_weight += grad_sampling_ptr; + const int grad_weight_stride = 1; + const int grad_loc_stride = 2; + const int qid_stride = num_heads * channels; + const int data_value_ptr_init_offset = b_col * spatial_size * qid_stride; + + for (int l_col=0; l_col < num_levels; ++l_col) + { + const int level_start_id = data_level_start_index[l_col]; + const int spatial_h_ptr = l_col << 1; + const int spatial_h = data_spatial_shapes[spatial_h_ptr]; + const int spatial_w = data_spatial_shapes[spatial_h_ptr + 1]; + const int value_ptr_offset = data_value_ptr_init_offset + level_start_id * qid_stride; + const scalar_t *data_value_ptr = data_value + value_ptr_offset; + scalar_t *grad_value_ptr = grad_value + value_ptr_offset; + + for (int p_col=0; p_col < num_point; ++p_col) + { + const scalar_t loc_w = data_sampling_loc[data_loc_w_ptr]; + const scalar_t loc_h = data_sampling_loc[data_loc_w_ptr + 1]; + const scalar_t weight = data_attn_weight[data_weight_ptr]; + + const scalar_t h_im = loc_h * spatial_h - 0.5; + const scalar_t w_im = loc_w * spatial_w - 0.5; + *(cache_grad_sampling_loc+(threadIdx.x << 1)) = 0; + *(cache_grad_sampling_loc+((threadIdx.x << 1) + 1)) = 0; + *(cache_grad_attn_weight+threadIdx.x)=0; + if (h_im > -1 && w_im > -1 && h_im < spatial_h && w_im < spatial_w) + { + ms_deform_attn_col2im_bilinear( + data_value_ptr, spatial_h, spatial_w, num_heads, channels, h_im, w_im, m_col, c_col, + top_grad, weight, grad_value_ptr, + cache_grad_sampling_loc+(threadIdx.x << 1), cache_grad_attn_weight+threadIdx.x); + } + + __syncthreads(); + if (tid == 0) + { + scalar_t _grad_w=cache_grad_sampling_loc[0], _grad_h=cache_grad_sampling_loc[1], _grad_a=cache_grad_attn_weight[0]; + int sid=2; + for (unsigned int tid = 1; tid < blockDim.x; ++tid) + { + _grad_w += cache_grad_sampling_loc[sid]; + _grad_h += cache_grad_sampling_loc[sid + 1]; + _grad_a += cache_grad_attn_weight[tid]; + sid += 2; + } + + + *grad_sampling_loc = _grad_w; + *(grad_sampling_loc + 1) = _grad_h; + *grad_attn_weight = _grad_a; + } + __syncthreads(); + + data_weight_ptr += 1; + data_loc_w_ptr += 2; + grad_attn_weight += grad_weight_stride; + grad_sampling_loc += grad_loc_stride; + } + } + } +} + +template +__global__ void ms_deformable_col2im_gpu_kernel_shm_reduce_v2(const int n, + const scalar_t *grad_col, + const scalar_t *data_value, + const int64_t *data_spatial_shapes, + const int64_t *data_level_start_index, + const scalar_t *data_sampling_loc, + const scalar_t *data_attn_weight, + const int batch_size, + const int spatial_size, + const int num_heads, + const int channels, + const int num_levels, + const int num_query, + const int num_point, + scalar_t *grad_value, + scalar_t *grad_sampling_loc, + scalar_t *grad_attn_weight) +{ + CUDA_KERNEL_LOOP(index, n) + { + extern __shared__ int _s[]; + scalar_t* cache_grad_sampling_loc = (scalar_t*)_s; + scalar_t* cache_grad_attn_weight = cache_grad_sampling_loc + 2 * blockDim.x; + unsigned int tid = threadIdx.x; + int _temp = index; + const int c_col = _temp % channels; + _temp /= channels; + const int sampling_index = _temp; + const int m_col = _temp % num_heads; + _temp /= num_heads; + const int q_col = _temp % num_query; + _temp /= num_query; + const int b_col = _temp; + + const scalar_t top_grad = grad_col[index]; + + int data_weight_ptr = sampling_index * num_levels * num_point; + int data_loc_w_ptr = data_weight_ptr << 1; + const int grad_sampling_ptr = data_weight_ptr; + grad_sampling_loc += grad_sampling_ptr << 1; + grad_attn_weight += grad_sampling_ptr; + const int grad_weight_stride = 1; + const int grad_loc_stride = 2; + const int qid_stride = num_heads * channels; + const int data_value_ptr_init_offset = b_col * spatial_size * qid_stride; + + for (int l_col=0; l_col < num_levels; ++l_col) + { + const int level_start_id = data_level_start_index[l_col]; + const int spatial_h_ptr = l_col << 1; + const int spatial_h = data_spatial_shapes[spatial_h_ptr]; + const int spatial_w = data_spatial_shapes[spatial_h_ptr + 1]; + const int value_ptr_offset = data_value_ptr_init_offset + level_start_id * qid_stride; + const scalar_t *data_value_ptr = data_value + value_ptr_offset; + scalar_t *grad_value_ptr = grad_value + value_ptr_offset; + + for (int p_col=0; p_col < num_point; ++p_col) + { + const scalar_t loc_w = data_sampling_loc[data_loc_w_ptr]; + const scalar_t loc_h = data_sampling_loc[data_loc_w_ptr + 1]; + const scalar_t weight = data_attn_weight[data_weight_ptr]; + + const scalar_t h_im = loc_h * spatial_h - 0.5; + const scalar_t w_im = loc_w * spatial_w - 0.5; + *(cache_grad_sampling_loc+(threadIdx.x << 1)) = 0; + *(cache_grad_sampling_loc+((threadIdx.x << 1) + 1)) = 0; + *(cache_grad_attn_weight+threadIdx.x)=0; + if (h_im > -1 && w_im > -1 && h_im < spatial_h && w_im < spatial_w) + { + ms_deform_attn_col2im_bilinear( + data_value_ptr, spatial_h, spatial_w, num_heads, channels, h_im, w_im, m_col, c_col, + top_grad, weight, grad_value_ptr, + cache_grad_sampling_loc+(threadIdx.x << 1), cache_grad_attn_weight+threadIdx.x); + } + + __syncthreads(); + + for (unsigned int s=blockDim.x/2, spre=blockDim.x; s>0; s>>=1, spre>>=1) + { + if (tid < s) { + const unsigned int xid1 = tid << 1; + const unsigned int xid2 = (tid + s) << 1; + cache_grad_attn_weight[tid] += cache_grad_attn_weight[tid + s]; + cache_grad_sampling_loc[xid1] += cache_grad_sampling_loc[xid2]; + cache_grad_sampling_loc[xid1 + 1] += cache_grad_sampling_loc[xid2 + 1]; + if (tid + (s << 1) < spre) + { + cache_grad_attn_weight[tid] += cache_grad_attn_weight[tid + (s << 1)]; + cache_grad_sampling_loc[xid1] += cache_grad_sampling_loc[xid2 + (s << 1)]; + cache_grad_sampling_loc[xid1 + 1] += cache_grad_sampling_loc[xid2 + 1 + (s << 1)]; + } + } + __syncthreads(); + } + + if (tid == 0) + { + *grad_sampling_loc = cache_grad_sampling_loc[0]; + *(grad_sampling_loc + 1) = cache_grad_sampling_loc[1]; + *grad_attn_weight = cache_grad_attn_weight[0]; + } + __syncthreads(); + + data_weight_ptr += 1; + data_loc_w_ptr += 2; + grad_attn_weight += grad_weight_stride; + grad_sampling_loc += grad_loc_stride; + } + } + } +} + +template +__global__ void ms_deformable_col2im_gpu_kernel_shm_reduce_v2_multi_blocks(const int n, + const scalar_t *grad_col, + const scalar_t *data_value, + const int64_t *data_spatial_shapes, + const int64_t *data_level_start_index, + const scalar_t *data_sampling_loc, + const scalar_t *data_attn_weight, + const int batch_size, + const int spatial_size, + const int num_heads, + const int channels, + const int num_levels, + const int num_query, + const int num_point, + scalar_t *grad_value, + scalar_t *grad_sampling_loc, + scalar_t *grad_attn_weight) +{ + CUDA_KERNEL_LOOP(index, n) + { + extern __shared__ int _s[]; + scalar_t* cache_grad_sampling_loc = (scalar_t*)_s; + scalar_t* cache_grad_attn_weight = cache_grad_sampling_loc + 2 * blockDim.x; + unsigned int tid = threadIdx.x; + int _temp = index; + const int c_col = _temp % channels; + _temp /= channels; + const int sampling_index = _temp; + const int m_col = _temp % num_heads; + _temp /= num_heads; + const int q_col = _temp % num_query; + _temp /= num_query; + const int b_col = _temp; + + const scalar_t top_grad = grad_col[index]; + + int data_weight_ptr = sampling_index * num_levels * num_point; + int data_loc_w_ptr = data_weight_ptr << 1; + const int grad_sampling_ptr = data_weight_ptr; + grad_sampling_loc += grad_sampling_ptr << 1; + grad_attn_weight += grad_sampling_ptr; + const int grad_weight_stride = 1; + const int grad_loc_stride = 2; + const int qid_stride = num_heads * channels; + const int data_value_ptr_init_offset = b_col * spatial_size * qid_stride; + + for (int l_col=0; l_col < num_levels; ++l_col) + { + const int level_start_id = data_level_start_index[l_col]; + const int spatial_h_ptr = l_col << 1; + const int spatial_h = data_spatial_shapes[spatial_h_ptr]; + const int spatial_w = data_spatial_shapes[spatial_h_ptr + 1]; + const int value_ptr_offset = data_value_ptr_init_offset + level_start_id * qid_stride; + const scalar_t *data_value_ptr = data_value + value_ptr_offset; + scalar_t *grad_value_ptr = grad_value + value_ptr_offset; + + for (int p_col=0; p_col < num_point; ++p_col) + { + const scalar_t loc_w = data_sampling_loc[data_loc_w_ptr]; + const scalar_t loc_h = data_sampling_loc[data_loc_w_ptr + 1]; + const scalar_t weight = data_attn_weight[data_weight_ptr]; + + const scalar_t h_im = loc_h * spatial_h - 0.5; + const scalar_t w_im = loc_w * spatial_w - 0.5; + *(cache_grad_sampling_loc+(threadIdx.x << 1)) = 0; + *(cache_grad_sampling_loc+((threadIdx.x << 1) + 1)) = 0; + *(cache_grad_attn_weight+threadIdx.x)=0; + if (h_im > -1 && w_im > -1 && h_im < spatial_h && w_im < spatial_w) + { + ms_deform_attn_col2im_bilinear( + data_value_ptr, spatial_h, spatial_w, num_heads, channels, h_im, w_im, m_col, c_col, + top_grad, weight, grad_value_ptr, + cache_grad_sampling_loc+(threadIdx.x << 1), cache_grad_attn_weight+threadIdx.x); + } + + __syncthreads(); + + for (unsigned int s=blockDim.x/2, spre=blockDim.x; s>0; s>>=1, spre>>=1) + { + if (tid < s) { + const unsigned int xid1 = tid << 1; + const unsigned int xid2 = (tid + s) << 1; + cache_grad_attn_weight[tid] += cache_grad_attn_weight[tid + s]; + cache_grad_sampling_loc[xid1] += cache_grad_sampling_loc[xid2]; + cache_grad_sampling_loc[xid1 + 1] += cache_grad_sampling_loc[xid2 + 1]; + if (tid + (s << 1) < spre) + { + cache_grad_attn_weight[tid] += cache_grad_attn_weight[tid + (s << 1)]; + cache_grad_sampling_loc[xid1] += cache_grad_sampling_loc[xid2 + (s << 1)]; + cache_grad_sampling_loc[xid1 + 1] += cache_grad_sampling_loc[xid2 + 1 + (s << 1)]; + } + } + __syncthreads(); + } + + if (tid == 0) + { + atomicAdd(grad_sampling_loc, cache_grad_sampling_loc[0]); + atomicAdd(grad_sampling_loc + 1, cache_grad_sampling_loc[1]); + atomicAdd(grad_attn_weight, cache_grad_attn_weight[0]); + } + __syncthreads(); + + data_weight_ptr += 1; + data_loc_w_ptr += 2; + grad_attn_weight += grad_weight_stride; + grad_sampling_loc += grad_loc_stride; + } + } + } +} + + +template +__global__ void ms_deformable_col2im_gpu_kernel_gm(const int n, + const scalar_t *grad_col, + const scalar_t *data_value, + const int64_t *data_spatial_shapes, + const int64_t *data_level_start_index, + const scalar_t *data_sampling_loc, + const scalar_t *data_attn_weight, + const int batch_size, + const int spatial_size, + const int num_heads, + const int channels, + const int num_levels, + const int num_query, + const int num_point, + scalar_t *grad_value, + scalar_t *grad_sampling_loc, + scalar_t *grad_attn_weight) +{ + CUDA_KERNEL_LOOP(index, n) + { + int _temp = index; + const int c_col = _temp % channels; + _temp /= channels; + const int sampling_index = _temp; + const int m_col = _temp % num_heads; + _temp /= num_heads; + const int q_col = _temp % num_query; + _temp /= num_query; + const int b_col = _temp; + + const scalar_t top_grad = grad_col[index]; + + int data_weight_ptr = sampling_index * num_levels * num_point; + int data_loc_w_ptr = data_weight_ptr << 1; + const int grad_sampling_ptr = data_weight_ptr; + grad_sampling_loc += grad_sampling_ptr << 1; + grad_attn_weight += grad_sampling_ptr; + const int grad_weight_stride = 1; + const int grad_loc_stride = 2; + const int qid_stride = num_heads * channels; + const int data_value_ptr_init_offset = b_col * spatial_size * qid_stride; + + for (int l_col=0; l_col < num_levels; ++l_col) + { + const int level_start_id = data_level_start_index[l_col]; + const int spatial_h_ptr = l_col << 1; + const int spatial_h = data_spatial_shapes[spatial_h_ptr]; + const int spatial_w = data_spatial_shapes[spatial_h_ptr + 1]; + const int value_ptr_offset = data_value_ptr_init_offset + level_start_id * qid_stride; + const scalar_t *data_value_ptr = data_value + value_ptr_offset; + scalar_t *grad_value_ptr = grad_value + value_ptr_offset; + + for (int p_col=0; p_col < num_point; ++p_col) + { + const scalar_t loc_w = data_sampling_loc[data_loc_w_ptr]; + const scalar_t loc_h = data_sampling_loc[data_loc_w_ptr + 1]; + const scalar_t weight = data_attn_weight[data_weight_ptr]; + + const scalar_t h_im = loc_h * spatial_h - 0.5; + const scalar_t w_im = loc_w * spatial_w - 0.5; + if (h_im > -1 && w_im > -1 && h_im < spatial_h && w_im < spatial_w) + { + ms_deform_attn_col2im_bilinear_gm( + data_value_ptr, spatial_h, spatial_w, num_heads, channels, h_im, w_im, m_col, c_col, + top_grad, weight, grad_value_ptr, + grad_sampling_loc, grad_attn_weight); + } + data_weight_ptr += 1; + data_loc_w_ptr += 2; + grad_attn_weight += grad_weight_stride; + grad_sampling_loc += grad_loc_stride; + } + } + } +} + + +template +void ms_deformable_im2col_cuda(cudaStream_t stream, + const scalar_t* data_value, + const int64_t* data_spatial_shapes, + const int64_t* data_level_start_index, + const scalar_t* data_sampling_loc, + const scalar_t* data_attn_weight, + const int batch_size, + const int spatial_size, + const int num_heads, + const int channels, + const int num_levels, + const int num_query, + const int num_point, + scalar_t* data_col) +{ + const int num_kernels = batch_size * num_query * num_heads * channels; + const int num_actual_kernels = batch_size * num_query * num_heads * channels; + const int num_threads = CUDA_NUM_THREADS; + ms_deformable_im2col_gpu_kernel + <<>>( + num_kernels, data_value, data_spatial_shapes, data_level_start_index, data_sampling_loc, data_attn_weight, + batch_size, spatial_size, num_heads, channels, num_levels, num_query, num_point, data_col); + + cudaError_t err = cudaGetLastError(); + if (err != cudaSuccess) + { + printf("error in ms_deformable_im2col_cuda: %s\n", cudaGetErrorString(err)); + } + +} + +template +void ms_deformable_col2im_cuda(cudaStream_t stream, + const scalar_t* grad_col, + const scalar_t* data_value, + const int64_t * data_spatial_shapes, + const int64_t * data_level_start_index, + const scalar_t * data_sampling_loc, + const scalar_t * data_attn_weight, + const int batch_size, + const int spatial_size, + const int num_heads, + const int channels, + const int num_levels, + const int num_query, + const int num_point, + scalar_t* grad_value, + scalar_t* grad_sampling_loc, + scalar_t* grad_attn_weight) +{ + const int num_threads = (channels > CUDA_NUM_THREADS)?CUDA_NUM_THREADS:channels; + const int num_kernels = batch_size * num_query * num_heads * channels; + const int num_actual_kernels = batch_size * num_query * num_heads * channels; + if (channels > 1024) + { + if ((channels & 1023) == 0) + { + ms_deformable_col2im_gpu_kernel_shm_reduce_v2_multi_blocks + <<>>( + num_kernels, + grad_col, + data_value, + data_spatial_shapes, + data_level_start_index, + data_sampling_loc, + data_attn_weight, + batch_size, + spatial_size, + num_heads, + channels, + num_levels, + num_query, + num_point, + grad_value, + grad_sampling_loc, + grad_attn_weight); + } + else + { + ms_deformable_col2im_gpu_kernel_gm + <<>>( + num_kernels, + grad_col, + data_value, + data_spatial_shapes, + data_level_start_index, + data_sampling_loc, + data_attn_weight, + batch_size, + spatial_size, + num_heads, + channels, + num_levels, + num_query, + num_point, + grad_value, + grad_sampling_loc, + grad_attn_weight); + } + } + else{ + switch(channels) + { + case 1: + ms_deformable_col2im_gpu_kernel_shm_blocksize_aware_reduce_v1 + <<>>( + num_kernels, + grad_col, + data_value, + data_spatial_shapes, + data_level_start_index, + data_sampling_loc, + data_attn_weight, + batch_size, + spatial_size, + num_heads, + channels, + num_levels, + num_query, + num_point, + grad_value, + grad_sampling_loc, + grad_attn_weight); + break; + case 2: + ms_deformable_col2im_gpu_kernel_shm_blocksize_aware_reduce_v1 + <<>>( + num_kernels, + grad_col, + data_value, + data_spatial_shapes, + data_level_start_index, + data_sampling_loc, + data_attn_weight, + batch_size, + spatial_size, + num_heads, + channels, + num_levels, + num_query, + num_point, + grad_value, + grad_sampling_loc, + grad_attn_weight); + break; + case 4: + ms_deformable_col2im_gpu_kernel_shm_blocksize_aware_reduce_v1 + <<>>( + num_kernels, + grad_col, + data_value, + data_spatial_shapes, + data_level_start_index, + data_sampling_loc, + data_attn_weight, + batch_size, + spatial_size, + num_heads, + channels, + num_levels, + num_query, + num_point, + grad_value, + grad_sampling_loc, + grad_attn_weight); + break; + case 8: + ms_deformable_col2im_gpu_kernel_shm_blocksize_aware_reduce_v1 + <<>>( + num_kernels, + grad_col, + data_value, + data_spatial_shapes, + data_level_start_index, + data_sampling_loc, + data_attn_weight, + batch_size, + spatial_size, + num_heads, + channels, + num_levels, + num_query, + num_point, + grad_value, + grad_sampling_loc, + grad_attn_weight); + break; + case 16: + ms_deformable_col2im_gpu_kernel_shm_blocksize_aware_reduce_v1 + <<>>( + num_kernels, + grad_col, + data_value, + data_spatial_shapes, + data_level_start_index, + data_sampling_loc, + data_attn_weight, + batch_size, + spatial_size, + num_heads, + channels, + num_levels, + num_query, + num_point, + grad_value, + grad_sampling_loc, + grad_attn_weight); + break; + case 32: + ms_deformable_col2im_gpu_kernel_shm_blocksize_aware_reduce_v1 + <<>>( + num_kernels, + grad_col, + data_value, + data_spatial_shapes, + data_level_start_index, + data_sampling_loc, + data_attn_weight, + batch_size, + spatial_size, + num_heads, + channels, + num_levels, + num_query, + num_point, + grad_value, + grad_sampling_loc, + grad_attn_weight); + break; + case 64: + ms_deformable_col2im_gpu_kernel_shm_blocksize_aware_reduce_v2 + <<>>( + num_kernels, + grad_col, + data_value, + data_spatial_shapes, + data_level_start_index, + data_sampling_loc, + data_attn_weight, + batch_size, + spatial_size, + num_heads, + channels, + num_levels, + num_query, + num_point, + grad_value, + grad_sampling_loc, + grad_attn_weight); + break; + case 128: + ms_deformable_col2im_gpu_kernel_shm_blocksize_aware_reduce_v2 + <<>>( + num_kernels, + grad_col, + data_value, + data_spatial_shapes, + data_level_start_index, + data_sampling_loc, + data_attn_weight, + batch_size, + spatial_size, + num_heads, + channels, + num_levels, + num_query, + num_point, + grad_value, + grad_sampling_loc, + grad_attn_weight); + break; + case 256: + ms_deformable_col2im_gpu_kernel_shm_blocksize_aware_reduce_v2 + <<>>( + num_kernels, + grad_col, + data_value, + data_spatial_shapes, + data_level_start_index, + data_sampling_loc, + data_attn_weight, + batch_size, + spatial_size, + num_heads, + channels, + num_levels, + num_query, + num_point, + grad_value, + grad_sampling_loc, + grad_attn_weight); + break; + case 512: + ms_deformable_col2im_gpu_kernel_shm_blocksize_aware_reduce_v2 + <<>>( + num_kernels, + grad_col, + data_value, + data_spatial_shapes, + data_level_start_index, + data_sampling_loc, + data_attn_weight, + batch_size, + spatial_size, + num_heads, + channels, + num_levels, + num_query, + num_point, + grad_value, + grad_sampling_loc, + grad_attn_weight); + break; + case 1024: + ms_deformable_col2im_gpu_kernel_shm_blocksize_aware_reduce_v2 + <<>>( + num_kernels, + grad_col, + data_value, + data_spatial_shapes, + data_level_start_index, + data_sampling_loc, + data_attn_weight, + batch_size, + spatial_size, + num_heads, + channels, + num_levels, + num_query, + num_point, + grad_value, + grad_sampling_loc, + grad_attn_weight); + break; + default: + if (channels < 64) + { + ms_deformable_col2im_gpu_kernel_shm_reduce_v1 + <<>>( + num_kernels, + grad_col, + data_value, + data_spatial_shapes, + data_level_start_index, + data_sampling_loc, + data_attn_weight, + batch_size, + spatial_size, + num_heads, + channels, + num_levels, + num_query, + num_point, + grad_value, + grad_sampling_loc, + grad_attn_weight); + } + else + { + ms_deformable_col2im_gpu_kernel_shm_reduce_v2 + <<>>( + num_kernels, + grad_col, + data_value, + data_spatial_shapes, + data_level_start_index, + data_sampling_loc, + data_attn_weight, + batch_size, + spatial_size, + num_heads, + channels, + num_levels, + num_query, + num_point, + grad_value, + grad_sampling_loc, + grad_attn_weight); + } + } + } + cudaError_t err = cudaGetLastError(); + if (err != cudaSuccess) + { + printf("error in ms_deformable_col2im_cuda: %s\n", cudaGetErrorString(err)); + } + +} \ No newline at end of file diff --git a/perception_models/apps/detection/DETA_pe/models/ops/src/ms_deform_attn.h b/perception_models/apps/detection/DETA_pe/models/ops/src/ms_deform_attn.h new file mode 100644 index 0000000000000000000000000000000000000000..ac0ef2ec25f7d0ee51ca2d807b159ddf85652017 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/models/ops/src/ms_deform_attn.h @@ -0,0 +1,62 @@ +/*! +************************************************************************************************** +* Deformable DETR +* Copyright (c) 2020 SenseTime. All Rights Reserved. +* Licensed under the Apache License, Version 2.0 [see LICENSE for details] +************************************************************************************************** +* Modified from https://github.com/chengdazhi/Deformable-Convolution-V2-PyTorch/tree/pytorch_1.0.0 +************************************************************************************************** +*/ + +#pragma once + +#include "cpu/ms_deform_attn_cpu.h" + +#ifdef WITH_CUDA +#include "cuda/ms_deform_attn_cuda.h" +#endif + + +at::Tensor +ms_deform_attn_forward( + const at::Tensor &value, + const at::Tensor &spatial_shapes, + const at::Tensor &level_start_index, + const at::Tensor &sampling_loc, + const at::Tensor &attn_weight, + const int im2col_step) +{ + if (value.type().is_cuda()) + { +#ifdef WITH_CUDA + return ms_deform_attn_cuda_forward( + value, spatial_shapes, level_start_index, sampling_loc, attn_weight, im2col_step); +#else + AT_ERROR("Not compiled with GPU support"); +#endif + } + AT_ERROR("Not implemented on the CPU"); +} + +std::vector +ms_deform_attn_backward( + const at::Tensor &value, + const at::Tensor &spatial_shapes, + const at::Tensor &level_start_index, + const at::Tensor &sampling_loc, + const at::Tensor &attn_weight, + const at::Tensor &grad_output, + const int im2col_step) +{ + if (value.type().is_cuda()) + { +#ifdef WITH_CUDA + return ms_deform_attn_cuda_backward( + value, spatial_shapes, level_start_index, sampling_loc, attn_weight, grad_output, im2col_step); +#else + AT_ERROR("Not compiled with GPU support"); +#endif + } + AT_ERROR("Not implemented on the CPU"); +} + diff --git a/perception_models/apps/detection/DETA_pe/models/ops/src/vision.cpp b/perception_models/apps/detection/DETA_pe/models/ops/src/vision.cpp new file mode 100644 index 0000000000000000000000000000000000000000..2201f63a51dca16d0b31148ed2c9e8e47ec15bdc --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/models/ops/src/vision.cpp @@ -0,0 +1,16 @@ +/*! +************************************************************************************************** +* Deformable DETR +* Copyright (c) 2020 SenseTime. All Rights Reserved. +* Licensed under the Apache License, Version 2.0 [see LICENSE for details] +************************************************************************************************** +* Modified from https://github.com/chengdazhi/Deformable-Convolution-V2-PyTorch/tree/pytorch_1.0.0 +************************************************************************************************** +*/ + +#include "ms_deform_attn.h" + +PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) { + m.def("ms_deform_attn_forward", &ms_deform_attn_forward, "ms_deform_attn_forward"); + m.def("ms_deform_attn_backward", &ms_deform_attn_backward, "ms_deform_attn_backward"); +} diff --git a/perception_models/apps/detection/DETA_pe/models/ops/test.py b/perception_models/apps/detection/DETA_pe/models/ops/test.py new file mode 100644 index 0000000000000000000000000000000000000000..8dbf6d5547d131f01a8c5c28b76557bd27a9334b --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/models/ops/test.py @@ -0,0 +1,89 @@ +# ------------------------------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------------------------------ +# Modified from https://github.com/chengdazhi/Deformable-Convolution-V2-PyTorch/tree/pytorch_1.0.0 +# ------------------------------------------------------------------------------------------------ + +from __future__ import absolute_import +from __future__ import print_function +from __future__ import division + +import time +import torch +import torch.nn as nn +from torch.autograd import gradcheck + +from functions.ms_deform_attn_func import MSDeformAttnFunction, ms_deform_attn_core_pytorch + + +N, M, D = 1, 2, 2 +Lq, L, P = 2, 2, 2 +shapes = torch.as_tensor([(6, 4), (3, 2)], dtype=torch.long).cuda() +level_start_index = torch.cat((shapes.new_zeros((1, )), shapes.prod(1).cumsum(0)[:-1])) +S = sum([(H*W).item() for H, W in shapes]) + + +torch.manual_seed(3) + + +@torch.no_grad() +def check_forward_equal_with_pytorch_double(): + value = torch.rand(N, S, M, D).cuda() * 0.01 + sampling_locations = torch.rand(N, Lq, M, L, P, 2).cuda() + attention_weights = torch.rand(N, Lq, M, L, P).cuda() + 1e-5 + attention_weights /= attention_weights.sum(-1, keepdim=True).sum(-2, keepdim=True) + im2col_step = 2 + output_pytorch = ms_deform_attn_core_pytorch(value.double(), shapes, sampling_locations.double(), attention_weights.double()).detach().cpu() + output_cuda = MSDeformAttnFunction.apply(value.double(), shapes, level_start_index, sampling_locations.double(), attention_weights.double(), im2col_step).detach().cpu() + fwdok = torch.allclose(output_cuda, output_pytorch) + max_abs_err = (output_cuda - output_pytorch).abs().max() + max_rel_err = ((output_cuda - output_pytorch).abs() / output_pytorch.abs()).max() + + print(f'* {fwdok} check_forward_equal_with_pytorch_double: max_abs_err {max_abs_err:.2e} max_rel_err {max_rel_err:.2e}') + + +@torch.no_grad() +def check_forward_equal_with_pytorch_float(): + value = torch.rand(N, S, M, D).cuda() * 0.01 + sampling_locations = torch.rand(N, Lq, M, L, P, 2).cuda() + attention_weights = torch.rand(N, Lq, M, L, P).cuda() + 1e-5 + attention_weights /= attention_weights.sum(-1, keepdim=True).sum(-2, keepdim=True) + im2col_step = 2 + output_pytorch = ms_deform_attn_core_pytorch(value, shapes, sampling_locations, attention_weights).detach().cpu() + output_cuda = MSDeformAttnFunction.apply(value, shapes, level_start_index, sampling_locations, attention_weights, im2col_step).detach().cpu() + fwdok = torch.allclose(output_cuda, output_pytorch, rtol=1e-2, atol=1e-3) + max_abs_err = (output_cuda - output_pytorch).abs().max() + max_rel_err = ((output_cuda - output_pytorch).abs() / output_pytorch.abs()).max() + + print(f'* {fwdok} check_forward_equal_with_pytorch_float: max_abs_err {max_abs_err:.2e} max_rel_err {max_rel_err:.2e}') + + +def check_gradient_numerical(channels=4, grad_value=True, grad_sampling_loc=True, grad_attn_weight=True): + + value = torch.rand(N, S, M, channels).cuda() * 0.01 + sampling_locations = torch.rand(N, Lq, M, L, P, 2).cuda() + attention_weights = torch.rand(N, Lq, M, L, P).cuda() + 1e-5 + attention_weights /= attention_weights.sum(-1, keepdim=True).sum(-2, keepdim=True) + im2col_step = 2 + func = MSDeformAttnFunction.apply + + value.requires_grad = grad_value + sampling_locations.requires_grad = grad_sampling_loc + attention_weights.requires_grad = grad_attn_weight + + gradok = gradcheck(func, (value.double(), shapes, level_start_index, sampling_locations.double(), attention_weights.double(), im2col_step)) + + print(f'* {gradok} check_gradient_numerical(D={channels})') + + +if __name__ == '__main__': + check_forward_equal_with_pytorch_double() + check_forward_equal_with_pytorch_float() + + for channels in [30, 32, 64, 71, 1025, 2048, 3096]: + check_gradient_numerical(channels, True, True, True) + + + diff --git a/perception_models/apps/detection/DETA_pe/models/pev1.py b/perception_models/apps/detection/DETA_pe/models/pev1.py new file mode 100644 index 0000000000000000000000000000000000000000..f480366d13234f0d2e5ab3e2c80050475a8710a0 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/models/pev1.py @@ -0,0 +1,686 @@ +import math +from collections import OrderedDict +from functools import partial +from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple, Union + +import torch +import torch.nn.functional as F +from einops import rearrange, repeat +from torch import broadcast_tensors, einsum, nn +from torch.nn.parameter import Parameter +from torch.utils.checkpoint import checkpoint + +from .utils_d2 import ( + add_decomposed_rel_pos, + PatchEmbed, + window_partition, + window_unpartition, +) + + +def get_abs_pos(abs_pos, has_cls_token, hw, tile=False): + h, w = hw + if has_cls_token: + abs_pos = abs_pos[:, 1:] + xy_num = abs_pos.shape[1] + size = int(math.sqrt(xy_num)) + assert size * size == xy_num + + if size != h or size != w: + if tile == True: + new_abs_pos = abs_pos.reshape(1, size, size, -1).tile( + [1, h // size + 1, w // size + 1, 1] + )[:, :h, :w, :] + + return new_abs_pos + else: + new_abs_pos = F.interpolate( + abs_pos.reshape(1, size, size, -1).permute(0, 3, 1, 2), + size=(h, w), + mode="bicubic", + align_corners=False, + ) + return new_abs_pos.permute(0, 2, 3, 1) + else: + return abs_pos.reshape(1, h, w, -1) + + +# broadcat, as tortoise-tts was using it +def broadcat(tensors, dim=-1): + broadcasted_tensors = broadcast_tensors(*tensors) + return torch.cat(broadcasted_tensors, dim=dim) + + +# rotary embedding helper functions +def rotate_half(x): + x = rearrange(x, "... (d r) -> ... d r", r=2) + x1, x2 = x.unbind(dim=-1) + x = torch.stack((-x2, x1), dim=-1) + return rearrange(x, "... d r -> ... (d r)") + + +class VisionRotaryEmbeddingFast(nn.Module): + def __init__( + self, + dim, + pt_seq_len=16, + ft_seq_len=None, + custom_freqs=None, + freqs_for="lang", + theta=10000, + max_freq=10, + num_freqs=1, + ): + super().__init__() + if custom_freqs: + freqs = custom_freqs + elif freqs_for == "lang": + freqs = 1.0 / ( + theta ** (torch.arange(0, dim, 2)[: (dim // 2)].float() / dim) + ) + elif freqs_for == "pixel": + freqs = torch.linspace(1.0, max_freq / 2, dim // 2) * pi + elif freqs_for == "constant": + freqs = torch.ones(num_freqs).float() + else: + raise ValueError(f"unknown modality {freqs_for}") + + if ft_seq_len is None: + ft_seq_len = pt_seq_len + t = ( + torch.arange(ft_seq_len) / ft_seq_len * pt_seq_len + 1 + ) # + 1 is hacking vev0 pt code + + freqs = torch.einsum("..., f -> ... f", t, freqs) + freqs = repeat(freqs, "... n -> ... (n r)", r=2) + # freqs = broadcat((freqs[:, None, :], freqs[None, :, :]), dim = -1) + freqs = broadcat( + (freqs[None, :, :], freqs[:, None, :]), dim=-1 + ) # follow vev0 pt code + + freqs_cos = freqs.cos().view(-1, freqs.shape[-1]) + freqs_sin = freqs.sin().view(-1, freqs.shape[-1]) + + self.register_buffer("freqs_cos", freqs_cos) + self.register_buffer("freqs_sin", freqs_sin) + + print("======== shape of rope freq", self.freqs_cos.shape, "========") + + def forward(self, tt): + return tt * self.freqs_cos + rotate_half(tt) * self.freqs_sin + + +class LayerNorm(nn.LayerNorm): + """Subclass torch's LayerNorm to handle fp16.""" + + def forward(self, x: torch.Tensor): + orig_type = x.dtype + # ret = super().forward(x.type(torch.float32)) + ret = F.layer_norm( + x.type(torch.float32), + self.normalized_shape, + self.weight.type(torch.float32), + self.bias.type(torch.float32), + self.eps, + ) + return ret.type(orig_type) + + +class QuickGELU(nn.Module): + def forward(self, x: torch.Tensor): + return x * torch.sigmoid(1.702 * x) + + +def drop_path( + x, drop_prob: float = 0.0, training: bool = False, scale_by_keep: bool = True +): + """Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). + + This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, + the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... + See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for + changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use + 'survival rate' as the argument. + + """ + if drop_prob == 0.0 or not training: + return x + keep_prob = 1 - drop_prob + shape = (x.shape[0],) + (1,) * ( + x.ndim - 1 + ) # work with diff dim tensors, not just 2D ConvNets + random_tensor = x.new_empty(shape).bernoulli_(keep_prob) + if keep_prob > 0.0 and scale_by_keep: + random_tensor.div_(keep_prob) + return x * random_tensor + + +class DropPath(nn.Module): + """Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).""" + + def __init__(self, drop_prob: float = 0.0, scale_by_keep: bool = True): + super(DropPath, self).__init__() + self.drop_prob = drop_prob + self.scale_by_keep = scale_by_keep + + def forward(self, x): + return drop_path(x, self.drop_prob, self.training, self.scale_by_keep) + + def extra_repr(self): + return f"drop_prob={round(self.drop_prob,3):0.3f}" + + +class Attention(nn.Module): + r""" + Implements attention based on Rope + """ + + def __init__( + self, + embed_dim: int, + num_heads: int, + dropout: float = 0.0, + bias: bool = True, + add_bias_kv: bool = False, + kdim: Optional[bool] = None, + vdim: Optional[bool] = None, + rope=None, + ): + super(Attention, self).__init__() + self.embed_dim = embed_dim + self.kdim = kdim if kdim is not None else embed_dim + self.vdim = vdim if vdim is not None else embed_dim + self._qkv_same_embed_dim = self.kdim == embed_dim and self.vdim == embed_dim + + self.num_heads = num_heads + self.dropout = dropout + self.head_dim = embed_dim // num_heads + assert ( + self.head_dim * num_heads == self.embed_dim + ), "embed_dim must be divisible by num_heads" + + if self._qkv_same_embed_dim is False: + self.q_proj_weight = Parameter(torch.Tensor(embed_dim, embed_dim)) + self.k_proj_weight = Parameter(torch.Tensor(embed_dim, self.kdim)) + self.v_proj_weight = Parameter(torch.Tensor(embed_dim, self.vdim)) + else: + self.in_proj_weight = Parameter(torch.empty(3 * embed_dim, embed_dim)) + + if bias: + self.in_proj_bias = Parameter(torch.empty(3 * embed_dim)) + else: + self.register_parameter("in_proj_bias", None) + self.out_proj = nn.Linear(embed_dim, embed_dim, bias=bias) + + if add_bias_kv: + self.bias_k = Parameter(torch.empty(1, 1, embed_dim)) + self.bias_v = Parameter(torch.empty(1, 1, embed_dim)) + else: + self.bias_k = self.bias_v = None + + self.rope = rope + + self.scale = self.head_dim ** (-0.5) + + def forward(self, query, attn_mask: Optional[torch.Tensor] = None): + batch, seq, embed_dim = query.shape + + proj = torch._C._nn.linear(query, self.in_proj_weight, self.in_proj_bias) + # reshape to 3, E and not E, 3 is deliberate for better memory coalescing and keeping same order as chunk() + proj = ( + proj.unflatten(-1, (3, embed_dim)) + .unsqueeze(0) + .transpose(0, -2) + .squeeze(-2) + .contiguous() + ) + q_, k_, v_ = proj[0], proj[1], proj[2] + + # Use "q_" so that we don't accidentally quit in pdb :) + q_ = rearrange(q_, "b s (h d) -> b h s d", h=self.num_heads) + k_ = rearrange(k_, "b s (h d) -> b h s d", h=self.num_heads) + v_ = rearrange(v_, "b s (h d) -> b h s d", h=self.num_heads) + + ## rope + q_ = self.rope(q_).type_as(v_) + k_ = self.rope(k_).type_as(v_) + + attn = (q_ * self.scale) @ k_.transpose(-2, -1) + attn = attn.softmax(dim=-1) + x_ = attn @ v_ + + x_ = rearrange(x_, "b h s d -> b s (h d)") + + return torch._C._nn.linear(x_, self.out_proj.weight, self.out_proj.bias) + + +class LayerScale(nn.Module): + def __init__( + self, + dim: int, + init_values: float = 1e-5, + inplace: bool = False, + ) -> None: + super().__init__() + self.inplace = inplace + self.gamma = nn.Parameter(init_values * torch.ones(dim)) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + return x.mul_(self.gamma) if self.inplace else x * self.gamma + + +class ResidualAttentionBlock(nn.Module): + def __init__( + self, + d_model: int, + n_head: int, + mlp_ratio=4.0, + act_layer=nn.GELU, + norm_layer=LayerNorm, + drop_path=0.0, + use_rel_pos=False, + rel_pos_zero_init=True, + window_size=0, + rope=None, + input_size=None, + attn_mask=None, + init_values=0.0, + ): + super().__init__() + + self.attn = Attention(embed_dim=d_model, num_heads=n_head, rope=rope) + self.ls_1 = ( + LayerScale(d_model, init_values=init_values) + if init_values > 0.0 + else nn.Identity() + ) + self.ln_1 = LayerNorm(d_model) + self.mlp = nn.Sequential( + OrderedDict( + [ + ("c_fc", nn.Linear(d_model, int(d_model * mlp_ratio))), + ("gelu", act_layer()), + ("c_proj", nn.Linear(int(d_model * mlp_ratio), d_model)), + ] + ) + ) + + self.drop_path = DropPath(drop_path) if drop_path > 0.0 else nn.Identity() + self.ln_2 = LayerNorm(d_model) + self.attn_mask = attn_mask + self.ls_2 = ( + LayerScale(d_model, init_values=init_values) + if init_values > 0.0 + else nn.Identity() + ) + self.window_size = window_size + + def attention_nhwc(self, x: torch.Tensor): + self.attn_mask = ( + self.attn_mask.to(dtype=x.dtype, device=x.device) + if self.attn_mask is not None + else None + ) + B, H, W, _ = x.shape + x = x.reshape(B, H * W, -1) + x = self.attn(x, attn_mask=self.attn_mask) + x = x.reshape(B, H, W, -1) + return x + + def forward(self, x: torch.Tensor): + shortcut = x + + x = self.ln_1(x) + # Window partition + if self.window_size > 0: + H, W = x.shape[1], x.shape[2] + x, pad_hw = window_partition(x, self.window_size) + + x = self.attention_nhwc(x) + # Reverse window partition + if self.window_size > 0: + x = window_unpartition(x, self.window_size, pad_hw, (H, W)) + + x = shortcut + self.drop_path(self.ls_1(x)) + x = x + self.drop_path(self.ls_2(self.mlp(self.ln_2(x)))) + return x + + +class Transformer(nn.Module): + def __init__( + self, + embed_dim: int, + depth: int, + num_heads: int, + mlp_ratio=4.0, + act_layer=nn.GELU, + norm_layer=LayerNorm, + drop_path_rate=0.0, + use_rel_pos=False, + rel_pos_zero_init=True, + window_size=0, + window_block_indexes=(), + img_size=1024, + patch_size=16, + rope_win=None, + rope_glb=None, + use_act_checkpoint=False, + act_checkpoint_ratio=1.0, + attn_mask=None, + init_values=0.0, + return_layer=[-1], + ): + super().__init__() + self.use_act_checkpoint = use_act_checkpoint + self.act_checkpoint_ratio = act_checkpoint_ratio + + # stochastic depth decay rule + dpr = [x.item() for x in torch.linspace(0, drop_path_rate, depth)] + + self.resblocks = nn.ModuleList() + for i in range(depth): + block = ResidualAttentionBlock( + embed_dim, + num_heads, + attn_mask=attn_mask, + drop_path=dpr[i], + mlp_ratio=mlp_ratio, + act_layer=act_layer, + norm_layer=norm_layer, + use_rel_pos=use_rel_pos, + rel_pos_zero_init=rel_pos_zero_init, + window_size=window_size if i in window_block_indexes else 0, + rope=rope_win if i in window_block_indexes else rope_glb, + input_size=(img_size // patch_size, img_size // patch_size), + init_values=init_values, + ) + self.resblocks.append(block) + + self.return_layer = return_layer + + def forward(self, x: torch.Tensor): + x_list = [] + for idx, blk in enumerate(self.resblocks): + if ( + self.use_act_checkpoint + and (idx / len(self.resblocks)) <= self.act_checkpoint_ratio + ): + x = checkpoint(blk, x) + else: + x = blk(x) + + if idx in self.return_layer or idx == len(self.resblocks) - 1: + x_list.append(x) + + return x, x_list + + +class PEv1_simpleFPN(nn.Module): + def __init__( + self, + img_size=1024, + patch_size=16, + in_chans=3, + embed_dim=768, + depth=12, + num_heads=12, + mlp_ratio=4.0, + qkv_bias=True, + drop_path_rate=0.0, + norm_layer=nn.LayerNorm, + act_layer=nn.GELU, + use_abs_pos=True, + use_rel_pos=False, + rel_pos_zero_init=True, + rope=True, + pt_hw_seq_len=16, + intp_freq=True, + window_size=0, + window_block_indexes=(), + residual_block_indexes=(), + use_act_checkpoint=False, + act_checkpoint_ratio=1.0, + pretrain_img_size=336, + pretrain_use_cls_token=True, + out_feature="last_feat", + tile_posemb=False, + init_values=0.0, + tta_rope=False, + return_layer=[-1], + ): + super().__init__() + self.pretrain_use_cls_token = pretrain_use_cls_token + + self.conv1 = nn.Conv2d( + in_channels=in_chans, + out_channels=embed_dim, + kernel_size=patch_size, + stride=patch_size, + bias=False, + ) + + if use_abs_pos: + # Initialize absolute positional embedding with pretrain image size. + num_patches = (pretrain_img_size // patch_size) * ( + pretrain_img_size // patch_size + ) + num_positions = (num_patches + 1) if pretrain_use_cls_token else num_patches + self.positional_embedding = nn.Parameter( + torch.zeros(1, num_positions, embed_dim) + ) + print("positional_embedding:", self.positional_embedding.shape) + print("positional_embedding:", self.positional_embedding.shape) + print("positional_embedding:", self.positional_embedding.shape) + + else: + self.positional_embedding = None + + self.tile_posemb = tile_posemb + + self.ln_pre = LayerNorm(embed_dim) + + half_head_dim = embed_dim // num_heads // 2 + hw_seq_len = img_size // patch_size + + self.rope_win = VisionRotaryEmbeddingFast( + dim=half_head_dim, + pt_seq_len=pt_hw_seq_len, + ft_seq_len=window_size if intp_freq else None, + ) + self.rope_glb = VisionRotaryEmbeddingFast( + dim=half_head_dim, + pt_seq_len=pt_hw_seq_len, + ft_seq_len=hw_seq_len if intp_freq else None, + ) + + self.transformer = Transformer( + embed_dim=embed_dim, + depth=depth, + num_heads=num_heads, + mlp_ratio=mlp_ratio, + act_layer=act_layer, + norm_layer=norm_layer, + drop_path_rate=drop_path_rate, + use_rel_pos=use_rel_pos, + rel_pos_zero_init=rel_pos_zero_init, + window_size=window_size, + window_block_indexes=window_block_indexes, + rope_win=self.rope_win, + rope_glb=self.rope_glb, + img_size=img_size, + patch_size=patch_size, + use_act_checkpoint=use_act_checkpoint, + act_checkpoint_ratio=act_checkpoint_ratio, + init_values=init_values, + return_layer=return_layer, + ) + + self._out_feature_channels = {out_feature: embed_dim} + self._out_feature_strides = {out_feature: patch_size} + self._out_features = [out_feature] + + if self.positional_embedding is not None: + nn.init.trunc_normal_(self.positional_embedding, std=0.02) + + self.return_layer = return_layer + # In our method, we don't use backbone feature with stride 4 + self.fpn1 = nn.Sequential( + nn.ConvTranspose2d(embed_dim, embed_dim // 2, kernel_size=2, stride=2), + ) + self.fpn2 = nn.Identity() + self.fpn3 = nn.MaxPool2d(kernel_size=2, stride=2) + + self.apply(self._init_weights) + + strides = [patch_size // 2, patch_size, patch_size * 2] + self._out_features = ["p{}".format(int(math.log2(s))) for s in strides] + self._out_feature_strides = { + "p3": 8, + "p4": 16, + "p5": 32, + } + self._out_feature_channels = { + "p3": embed_dim // 2, + "p4": embed_dim, + "p5": embed_dim, + } + self._size_divisibility = strides[-1] + self._square_pad = img_size + + def _init_weights(self, m): + if isinstance(m, nn.Linear): + nn.init.trunc_normal_(m.weight, std=0.02) + if isinstance(m, nn.Linear) and m.bias is not None: + nn.init.constant_(m.bias, 0) + elif isinstance(m, nn.LayerNorm): + nn.init.constant_(m.bias, 0) + nn.init.constant_(m.weight, 1.0) + + def forward(self, x): + x = self.conv1(x) + x = x.permute(0, 2, 3, 1) + + if self.positional_embedding is not None: + x = x + get_abs_pos( + self.positional_embedding, + self.pretrain_use_cls_token, + (x.shape[1], x.shape[2]), + self.tile_posemb, + ) + x = self.ln_pre(x) + + x, x_list = self.transformer(x) + + xp = x.permute(0, 3, 1, 2) # (b, h, w, c) --> (b, c, h, w) + + features = [] + ops = [self.fpn1, self.fpn2, self.fpn3] + for i in range(len(ops)): + features.append(ops[i](xp)) + rets = {"p{}".format(u + 3): v for (u, v) in enumerate(features)} + + return rets + + +def get_pev1_and_fpn_backbone(args): + if args.lsj_img_size_max > 0: + img_size = args.lsj_img_size_max + else: + img_size = args.lsj_img_size + use_act_checkpoint = args.backbone_use_act_checkpoint + act_checkpoint_ratio = args.backbone_act_checkpoint_ratio + init_values = args.backbone_init_values + tile_posemb = args.backbone_tile_posemb + tta_rope = args.backbone_tta_rope + multi_layer = args.backbone_multi_layer + backbone_dp = args.backbone_dp + + if args.backbone_size == "G": + embed_dim, depth, num_heads, mlp_ratio, dp = 1536, 50, 16, 8960 / 1536, 0.5 + pretrain_img_size, patch_size, window_size = 224, 16, 14 + window_block_indexes = ( + list(range(0, 12)) + + list(range(13, 24)) + + list(range(25, 36)) + + list(range(37, 49)) + ) + pretrain_use_cls_token = False + if multi_layer: + return_layer = [12, 24, 36, 49] + else: + return_layer = [-1] + + elif args.backbone_size == "Gwin384": + embed_dim, depth, num_heads, mlp_ratio, dp = 1536, 50, 16, 8960 / 1536, 0.5 + pretrain_img_size, patch_size, window_size = 384, 16, 24 + window_block_indexes = ( + list(range(0, 12)) + + list(range(13, 24)) + + list(range(25, 36)) + + list(range(37, 49)) + ) + pretrain_use_cls_token = False + if multi_layer: + return_layer = [12, 24, 36, 49] + else: + return_layer = [-1] + + elif args.backbone_size == "Gwin512": + embed_dim, depth, num_heads, mlp_ratio, dp = 1536, 50, 16, 8960 / 1536, 0.5 + pretrain_img_size, patch_size, window_size = 512, 16, 32 + window_block_indexes = ( + list(range(0, 12)) + + list(range(13, 24)) + + list(range(25, 36)) + + list(range(37, 49)) + ) + pretrain_use_cls_token = False + if multi_layer: + return_layer = [12, 24, 36, 49] + else: + return_layer = [-1] + else: + raise ValueError("Unsupported backbone size") + + if backbone_dp >= 0: + dp = backbone_dp + + assert ( + depth == args.backbone_layers + ), f"backbone depth {depth} and layers {args.backbone_layers}(from config) must be the same" + + model = PEv1_simpleFPN( + use_act_checkpoint=use_act_checkpoint, + act_checkpoint_ratio=act_checkpoint_ratio, + pretrain_img_size=pretrain_img_size, + pretrain_use_cls_token=pretrain_use_cls_token, + img_size=img_size, + patch_size=patch_size, + embed_dim=embed_dim, + depth=depth, + num_heads=num_heads, + drop_path_rate=dp, + window_size=window_size, + pt_hw_seq_len=16, # Maybe a bug ? + mlp_ratio=mlp_ratio, + qkv_bias=True, + norm_layer=partial(nn.LayerNorm, eps=1e-6), + window_block_indexes=window_block_indexes, + residual_block_indexes=[], + use_rel_pos=True, + out_feature="last_feat", + tile_posemb=tile_posemb, + init_values=init_values, + tta_rope=tta_rope, + return_layer=return_layer, + ) + + pretrained_backbone_path = args.backbone_path + if pretrained_backbone_path: + state_dict = torch.load(pretrained_backbone_path, map_location="cpu") + load_info = model.load_state_dict(state_dict["model"], strict=False) + print("Missing keys", load_info.missing_keys) + print("Unexpected keys", load_info.unexpected_keys) + else: + print("Skip pretrained backbone loading") + return model diff --git a/perception_models/apps/detection/DETA_pe/models/position_encoding.py b/perception_models/apps/detection/DETA_pe/models/position_encoding.py new file mode 100644 index 0000000000000000000000000000000000000000..a92f0d36aec69ed8d918de0214bae57d335aeb10 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/models/position_encoding.py @@ -0,0 +1,97 @@ +# ------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------ +# Modified from DETR (https://github.com/facebookresearch/detr) +# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved +# ------------------------------------------------------------------------ + +""" +Various positional encodings for the transformer. +""" +import math +import torch +from torch import nn + +from util.misc import NestedTensor + + +class PositionEmbeddingSine(nn.Module): + """ + This is a more standard version of the position embedding, very similar to the one + used by the Attention is all you need paper, generalized to work on images. + """ + def __init__(self, num_pos_feats=64, temperature=10000, normalize=False, scale=None): + super().__init__() + self.num_pos_feats = num_pos_feats + self.temperature = temperature + self.normalize = normalize + if scale is not None and normalize is False: + raise ValueError("normalize should be True if scale is passed") + if scale is None: + scale = 2 * math.pi + self.scale = scale + + def forward(self, tensor_list: NestedTensor): + x = tensor_list.tensors + mask = tensor_list.mask + assert mask is not None + not_mask = ~mask + y_embed = not_mask.cumsum(1, dtype=torch.float32) + x_embed = not_mask.cumsum(2, dtype=torch.float32) + if self.normalize: + eps = 1e-6 + y_embed = (y_embed - 0.5) / (y_embed[:, -1:, :] + eps) * self.scale + x_embed = (x_embed - 0.5) / (x_embed[:, :, -1:] + eps) * self.scale + + dim_t = torch.arange(self.num_pos_feats, dtype=torch.float32, device=x.device) + dim_t = self.temperature ** (2 * (dim_t // 2) / self.num_pos_feats) + + pos_x = x_embed[:, :, :, None] / dim_t + pos_y = y_embed[:, :, :, None] / dim_t + pos_x = torch.stack((pos_x[:, :, :, 0::2].sin(), pos_x[:, :, :, 1::2].cos()), dim=4).flatten(3) + pos_y = torch.stack((pos_y[:, :, :, 0::2].sin(), pos_y[:, :, :, 1::2].cos()), dim=4).flatten(3) + pos = torch.cat((pos_y, pos_x), dim=3).permute(0, 3, 1, 2) + return pos + + +class PositionEmbeddingLearned(nn.Module): + """ + Absolute pos embedding, learned. + """ + def __init__(self, num_pos_feats=256): + super().__init__() + self.row_embed = nn.Embedding(50, num_pos_feats) + self.col_embed = nn.Embedding(50, num_pos_feats) + self.reset_parameters() + + def reset_parameters(self): + nn.init.uniform_(self.row_embed.weight) + nn.init.uniform_(self.col_embed.weight) + + def forward(self, tensor_list: NestedTensor): + x = tensor_list.tensors + h, w = x.shape[-2:] + i = torch.arange(w, device=x.device) + j = torch.arange(h, device=x.device) + x_emb = self.col_embed(i) + y_emb = self.row_embed(j) + pos = torch.cat([ + x_emb.unsqueeze(0).repeat(h, 1, 1), + y_emb.unsqueeze(1).repeat(1, w, 1), + ], dim=-1).permute(2, 0, 1).unsqueeze(0).repeat(x.shape[0], 1, 1, 1) + return pos + + +def build_position_encoding(args): + N_steps = args.hidden_dim // 2 + if args.position_embedding in ('v2', 'sine'): + # TODO find a better way of exposing other arguments + position_embedding = PositionEmbeddingSine(N_steps, normalize=True) + elif args.position_embedding in ('v3', 'learned'): + position_embedding = PositionEmbeddingLearned(N_steps) + else: + raise ValueError(f"not supported {args.position_embedding}") + + return position_embedding diff --git a/perception_models/apps/detection/DETA_pe/models/segmentation.py b/perception_models/apps/detection/DETA_pe/models/segmentation.py new file mode 100644 index 0000000000000000000000000000000000000000..c801c0eaada3df0f286f6288266a00053b82472e --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/models/segmentation.py @@ -0,0 +1,369 @@ +# ------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------ +# Modified from DETR (https://github.com/facebookresearch/detr) +# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved +# ------------------------------------------------------------------------ + +""" +This file provides the definition of the convolutional heads used to predict masks, as well as the losses +""" +import io +from collections import defaultdict + +import torch +import torch.nn as nn +import torch.nn.functional as F +from PIL import Image + +import util.box_ops as box_ops +from util.misc import NestedTensor, interpolate, nested_tensor_from_tensor_list + +try: + from panopticapi.utils import id2rgb, rgb2id +except ImportError: + pass + + +class DETRsegm(nn.Module): + def __init__(self, detr, freeze_detr=False): + super().__init__() + self.detr = detr + + if freeze_detr: + for p in self.parameters(): + p.requires_grad_(False) + + hidden_dim, nheads = detr.transformer.d_model, detr.transformer.nhead + self.bbox_attention = MHAttentionMap(hidden_dim, hidden_dim, nheads, dropout=0) + self.mask_head = MaskHeadSmallConv(hidden_dim + nheads, [1024, 512, 256], hidden_dim) + + def forward(self, samples: NestedTensor): + if not isinstance(samples, NestedTensor): + samples = nested_tensor_from_tensor_list(samples) + features, pos = self.detr.backbone(samples) + + bs = features[-1].tensors.shape[0] + + src, mask = features[-1].decompose() + src_proj = self.detr.input_proj(src) + hs, memory = self.detr.transformer(src_proj, mask, self.detr.query_embed.weight, pos[-1]) + + outputs_class = self.detr.class_embed(hs) + outputs_coord = self.detr.bbox_embed(hs).sigmoid() + out = {"pred_logits": outputs_class[-1], "pred_boxes": outputs_coord[-1]} + if self.detr.aux_loss: + out["aux_outputs"] = [ + {"pred_logits": a, "pred_boxes": b} for a, b in zip(outputs_class[:-1], outputs_coord[:-1]) + ] + + # FIXME h_boxes takes the last one computed, keep this in mind + bbox_mask = self.bbox_attention(hs[-1], memory, mask=mask) + + seg_masks = self.mask_head(src_proj, bbox_mask, [features[2].tensors, features[1].tensors, features[0].tensors]) + outputs_seg_masks = seg_masks.view(bs, self.detr.num_queries, seg_masks.shape[-2], seg_masks.shape[-1]) + + out["pred_masks"] = outputs_seg_masks + return out + + +class MaskHeadSmallConv(nn.Module): + """ + Simple convolutional head, using group norm. + Upsampling is done using a FPN approach + """ + + def __init__(self, dim, fpn_dims, context_dim): + super().__init__() + + inter_dims = [dim, context_dim // 2, context_dim // 4, context_dim // 8, context_dim // 16, context_dim // 64] + self.lay1 = torch.nn.Conv2d(dim, dim, 3, padding=1) + self.gn1 = torch.nn.GroupNorm(8, dim) + self.lay2 = torch.nn.Conv2d(dim, inter_dims[1], 3, padding=1) + self.gn2 = torch.nn.GroupNorm(8, inter_dims[1]) + self.lay3 = torch.nn.Conv2d(inter_dims[1], inter_dims[2], 3, padding=1) + self.gn3 = torch.nn.GroupNorm(8, inter_dims[2]) + self.lay4 = torch.nn.Conv2d(inter_dims[2], inter_dims[3], 3, padding=1) + self.gn4 = torch.nn.GroupNorm(8, inter_dims[3]) + self.lay5 = torch.nn.Conv2d(inter_dims[3], inter_dims[4], 3, padding=1) + self.gn5 = torch.nn.GroupNorm(8, inter_dims[4]) + self.out_lay = torch.nn.Conv2d(inter_dims[4], 1, 3, padding=1) + + self.dim = dim + + self.adapter1 = torch.nn.Conv2d(fpn_dims[0], inter_dims[1], 1) + self.adapter2 = torch.nn.Conv2d(fpn_dims[1], inter_dims[2], 1) + self.adapter3 = torch.nn.Conv2d(fpn_dims[2], inter_dims[3], 1) + + for m in self.modules(): + if isinstance(m, nn.Conv2d): + nn.init.kaiming_uniform_(m.weight, a=1) + nn.init.constant_(m.bias, 0) + + def forward(self, x, bbox_mask, fpns): + def expand(tensor, length): + return tensor.unsqueeze(1).repeat(1, int(length), 1, 1, 1).flatten(0, 1) + + x = torch.cat([expand(x, bbox_mask.shape[1]), bbox_mask.flatten(0, 1)], 1) + + x = self.lay1(x) + x = self.gn1(x) + x = F.relu(x) + x = self.lay2(x) + x = self.gn2(x) + x = F.relu(x) + + cur_fpn = self.adapter1(fpns[0]) + if cur_fpn.size(0) != x.size(0): + cur_fpn = expand(cur_fpn, x.size(0) / cur_fpn.size(0)) + x = cur_fpn + F.interpolate(x, size=cur_fpn.shape[-2:], mode="nearest") + x = self.lay3(x) + x = self.gn3(x) + x = F.relu(x) + + cur_fpn = self.adapter2(fpns[1]) + if cur_fpn.size(0) != x.size(0): + cur_fpn = expand(cur_fpn, x.size(0) / cur_fpn.size(0)) + x = cur_fpn + F.interpolate(x, size=cur_fpn.shape[-2:], mode="nearest") + x = self.lay4(x) + x = self.gn4(x) + x = F.relu(x) + + cur_fpn = self.adapter3(fpns[2]) + if cur_fpn.size(0) != x.size(0): + cur_fpn = expand(cur_fpn, x.size(0) / cur_fpn.size(0)) + x = cur_fpn + F.interpolate(x, size=cur_fpn.shape[-2:], mode="nearest") + x = self.lay5(x) + x = self.gn5(x) + x = F.relu(x) + + x = self.out_lay(x) + return x + + +class MHAttentionMap(nn.Module): + """This is a 2D attention module, which only returns the attention softmax (no multiplication by value)""" + + def __init__(self, query_dim, hidden_dim, num_heads, dropout=0, bias=True): + super().__init__() + self.num_heads = num_heads + self.hidden_dim = hidden_dim + self.dropout = nn.Dropout(dropout) + + self.q_linear = nn.Linear(query_dim, hidden_dim, bias=bias) + self.k_linear = nn.Linear(query_dim, hidden_dim, bias=bias) + + nn.init.zeros_(self.k_linear.bias) + nn.init.zeros_(self.q_linear.bias) + nn.init.xavier_uniform_(self.k_linear.weight) + nn.init.xavier_uniform_(self.q_linear.weight) + self.normalize_fact = float(hidden_dim / self.num_heads) ** -0.5 + + def forward(self, q, k, mask=None): + q = self.q_linear(q) + k = F.conv2d(k, self.k_linear.weight.unsqueeze(-1).unsqueeze(-1), self.k_linear.bias) + qh = q.view(q.shape[0], q.shape[1], self.num_heads, self.hidden_dim // self.num_heads) + kh = k.view(k.shape[0], self.num_heads, self.hidden_dim // self.num_heads, k.shape[-2], k.shape[-1]) + weights = torch.einsum("bqnc,bnchw->bqnhw", qh * self.normalize_fact, kh) + + if mask is not None: + weights.masked_fill_(mask.unsqueeze(1).unsqueeze(1), float("-inf")) + weights = F.softmax(weights.flatten(2), dim=-1).view_as(weights) + weights = self.dropout(weights) + return weights + + +def dice_loss(inputs, targets, num_boxes): + """ + Compute the DICE loss, similar to generalized IOU for masks + Args: + inputs: A float tensor of arbitrary shape. + The predictions for each example. + targets: A float tensor with the same shape as inputs. Stores the binary + classification label for each element in inputs + (0 for the negative class and 1 for the positive class). + """ + inputs = inputs.sigmoid() + inputs = inputs.flatten(1) + numerator = 2 * (inputs * targets).sum(1) + denominator = inputs.sum(-1) + targets.sum(-1) + loss = 1 - (numerator + 1) / (denominator + 1) + return loss.sum() / num_boxes + + +def sigmoid_focal_loss(inputs, targets, num_boxes, alpha: float = 0.25, gamma: float = 2): + """ + Loss used in RetinaNet for dense detection: https://arxiv.org/abs/1708.02002. + Args: + inputs: A float tensor of arbitrary shape. + The predictions for each example. + targets: A float tensor with the same shape as inputs. Stores the binary + classification label for each element in inputs + (0 for the negative class and 1 for the positive class). + alpha: (optional) Weighting factor in range (0,1) to balance + positive vs negative examples. Default = -1 (no weighting). + gamma: Exponent of the modulating factor (1 - p_t) to + balance easy vs hard examples. + Returns: + Loss tensor + """ + prob = inputs.sigmoid() + ce_loss = F.binary_cross_entropy_with_logits(inputs, targets, reduction="none") + p_t = prob * targets + (1 - prob) * (1 - targets) + loss = ce_loss * ((1 - p_t) ** gamma) + + if alpha >= 0: + alpha_t = alpha * targets + (1 - alpha) * (1 - targets) + loss = alpha_t * loss + + return loss.mean(1).sum() / num_boxes + + +class PostProcessSegm(nn.Module): + def __init__(self, threshold=0.5): + super().__init__() + self.threshold = threshold + + @torch.no_grad() + def forward(self, results, outputs, orig_target_sizes, max_target_sizes): + assert len(orig_target_sizes) == len(max_target_sizes) + max_h, max_w = max_target_sizes.max(0)[0].tolist() + outputs_masks = outputs["pred_masks"].squeeze(2) + outputs_masks = F.interpolate(outputs_masks, size=(max_h, max_w), mode="bilinear", align_corners=False) + outputs_masks = (outputs_masks.sigmoid() > self.threshold).cpu() + + for i, (cur_mask, t, tt) in enumerate(zip(outputs_masks, max_target_sizes, orig_target_sizes)): + img_h, img_w = t[0], t[1] + results[i]["masks"] = cur_mask[:, :img_h, :img_w].unsqueeze(1) + results[i]["masks"] = F.interpolate( + results[i]["masks"].float(), size=tuple(tt.tolist()), mode="nearest" + ).byte() + + return results + + +class PostProcessPanoptic(nn.Module): + """This class converts the output of the model to the final panoptic result, in the format expected by the + coco panoptic API """ + + def __init__(self, is_thing_map, threshold=0.85): + """ + Parameters: + is_thing_map: This is a whose keys are the class ids, and the values a boolean indicating whether + the class is a thing (True) or a stuff (False) class + threshold: confidence threshold: segments with confidence lower than this will be deleted + """ + super().__init__() + self.threshold = threshold + self.is_thing_map = is_thing_map + + def forward(self, outputs, processed_sizes, target_sizes=None): + """ This function computes the panoptic prediction from the model's predictions. + Parameters: + outputs: This is a dict coming directly from the model. See the model doc for the content. + processed_sizes: This is a list of tuples (or torch tensors) of sizes of the images that were passed to the + model, ie the size after data augmentation but before batching. + target_sizes: This is a list of tuples (or torch tensors) corresponding to the requested final size + of each prediction. If left to None, it will default to the processed_sizes + """ + if target_sizes is None: + target_sizes = processed_sizes + assert len(processed_sizes) == len(target_sizes) + out_logits, raw_masks, raw_boxes = outputs["pred_logits"], outputs["pred_masks"], outputs["pred_boxes"] + assert len(out_logits) == len(raw_masks) == len(target_sizes) + preds = [] + + def to_tuple(tup): + if isinstance(tup, tuple): + return tup + return tuple(tup.cpu().tolist()) + + for cur_logits, cur_masks, cur_boxes, size, target_size in zip( + out_logits, raw_masks, raw_boxes, processed_sizes, target_sizes + ): + # we filter empty queries and detection below threshold + scores, labels = cur_logits.softmax(-1).max(-1) + keep = labels.ne(outputs["pred_logits"].shape[-1] - 1) & (scores > self.threshold) + cur_scores, cur_classes = cur_logits.softmax(-1).max(-1) + cur_scores = cur_scores[keep] + cur_classes = cur_classes[keep] + cur_masks = cur_masks[keep] + cur_masks = interpolate(cur_masks[None], to_tuple(size), mode="bilinear").squeeze(0) + cur_boxes = box_ops.box_cxcywh_to_xyxy(cur_boxes[keep]) + + h, w = cur_masks.shape[-2:] + assert len(cur_boxes) == len(cur_classes) + + # It may be that we have several predicted masks for the same stuff class. + # In the following, we track the list of masks ids for each stuff class (they are merged later on) + cur_masks = cur_masks.flatten(1) + stuff_equiv_classes = defaultdict(lambda: []) + for k, label in enumerate(cur_classes): + if not self.is_thing_map[label.item()]: + stuff_equiv_classes[label.item()].append(k) + + def get_ids_area(masks, scores, dedup=False): + # This helper function creates the final panoptic segmentation image + # It also returns the area of the masks that appears on the image + + m_id = masks.transpose(0, 1).softmax(-1) + + if m_id.shape[-1] == 0: + # We didn't detect any mask :( + m_id = torch.zeros((h, w), dtype=torch.long, device=m_id.device) + else: + m_id = m_id.argmax(-1).view(h, w) + + if dedup: + # Merge the masks corresponding to the same stuff class + for equiv in stuff_equiv_classes.values(): + if len(equiv) > 1: + for eq_id in equiv: + m_id.masked_fill_(m_id.eq(eq_id), equiv[0]) + + final_h, final_w = to_tuple(target_size) + + seg_img = Image.fromarray(id2rgb(m_id.view(h, w).cpu().numpy())) + seg_img = seg_img.resize(size=(final_w, final_h), resample=Image.NEAREST) + + np_seg_img = ( + torch.ByteTensor(torch.ByteStorage.from_buffer(seg_img.tobytes())).view(final_h, final_w, 3).numpy() + ) + m_id = torch.from_numpy(rgb2id(np_seg_img)) + + area = [] + for i in range(len(scores)): + area.append(m_id.eq(i).sum().item()) + return area, seg_img + + area, seg_img = get_ids_area(cur_masks, cur_scores, dedup=True) + if cur_classes.numel() > 0: + # We know filter empty masks as long as we find some + while True: + filtered_small = torch.as_tensor( + [area[i] <= 4 for i, c in enumerate(cur_classes)], dtype=torch.bool, device=keep.device + ) + if filtered_small.any().item(): + cur_scores = cur_scores[~filtered_small] + cur_classes = cur_classes[~filtered_small] + cur_masks = cur_masks[~filtered_small] + area, seg_img = get_ids_area(cur_masks, cur_scores) + else: + break + + else: + cur_classes = torch.ones(1, dtype=torch.long, device=cur_classes.device) + + segments_info = [] + for i, a in enumerate(area): + cat = cur_classes[i].item() + segments_info.append({"id": i, "isthing": self.is_thing_map[cat], "category_id": cat, "area": a}) + del cur_classes + + with io.BytesIO() as out: + seg_img.save(out, format="PNG") + predictions = {"png_string": out.getvalue(), "segments_info": segments_info} + preds.append(predictions) + return preds diff --git a/perception_models/apps/detection/DETA_pe/models/swin.py b/perception_models/apps/detection/DETA_pe/models/swin.py new file mode 100644 index 0000000000000000000000000000000000000000..51dbb592a3647d09c0eccfbee283e247f2d6ec83 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/models/swin.py @@ -0,0 +1,801 @@ +# -------------------------------------------------------- +# Swin Transformer +# Copyright (c) 2021 Microsoft +# Licensed under The MIT License [see LICENSE for details] +# Written by Ze Liu, Yutong Lin, Yixuan Wei +# -------------------------------------------------------- + +# Copyright (c) Facebook, Inc. and its affiliates. +# Modified by Bowen Cheng from https://github.com/SwinTransformer/Swin-Transformer-Semantic-Segmentation/blob/main/mmseg/models/backbones/swin_transformer.py +# Modified by Jeffrey Ouyang-Zhang +import collections.abc +from itertools import repeat + +import numpy as np +import torch +import torch.nn as nn +import torch.nn.functional as F +import torch.utils.checkpoint as checkpoint + +# from timm.models.layers import DropPath, to_2tuple, trunc_normal_ + +swin_l_kwargs = { + "pretrain_img_size": 384, + "embed_dim": 192, + "depths": [2, 2, 18, 2], + "num_heads": [6, 12, 24, 48], + "window_size": 12, + "ape": False, + "drop_path_rate": 0.3, + "patch_norm": True, + "out_indices": (1, 2, 3), + "use_checkpoint": True, +} +swin_l_weights = "/checkpoint/onevision/peizesun/public_model/swin/swin_large_patch4_window12_384_22k.pth" + + +def drop_path( + x, drop_prob: float = 0.0, training: bool = False, scale_by_keep: bool = True +): + """Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). + + This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, + the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... + See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for + changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use + 'survival rate' as the argument. + + """ + if drop_prob == 0.0 or not training: + return x + keep_prob = 1 - drop_prob + shape = (x.shape[0],) + (1,) * ( + x.ndim - 1 + ) # work with diff dim tensors, not just 2D ConvNets + random_tensor = x.new_empty(shape).bernoulli_(keep_prob) + if keep_prob > 0.0 and scale_by_keep: + random_tensor.div_(keep_prob) + return x * random_tensor + + +class DropPath(nn.Module): + """Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).""" + + def __init__(self, drop_prob: float = 0.0, scale_by_keep: bool = True): + super(DropPath, self).__init__() + self.drop_prob = drop_prob + self.scale_by_keep = scale_by_keep + + def forward(self, x): + return drop_path(x, self.drop_prob, self.training, self.scale_by_keep) + + def extra_repr(self): + return f"drop_prob={round(self.drop_prob,3):0.3f}" + + +# From PyTorch internals +def _ntuple(n): + def parse(x): + if isinstance(x, collections.abc.Iterable) and not isinstance(x, str): + return tuple(x) + return tuple(repeat(x, n)) + + return parse + + +to_1tuple = _ntuple(1) +to_2tuple = _ntuple(2) + + +def get_swinl(**add_kwargs): + model = SwinTransformer(**swin_l_kwargs, **add_kwargs) + state_dict = torch.load(swin_l_weights) + load_info = model.load_state_dict(state_dict["model"], strict=False) + print("Missing swin keys", load_info.missing_keys) + print("Unexpected swin keys", load_info.unexpected_keys) + return model + + +class Mlp(nn.Module): + """Multilayer perceptron.""" + + def __init__( + self, + in_features, + hidden_features=None, + out_features=None, + act_layer=nn.GELU, + drop=0.0, + ): + super().__init__() + out_features = out_features or in_features + hidden_features = hidden_features or in_features + self.fc1 = nn.Linear(in_features, hidden_features) + self.act = act_layer() + self.fc2 = nn.Linear(hidden_features, out_features) + self.drop = nn.Dropout(drop) + + def forward(self, x): + x = self.fc1(x) + x = self.act(x) + x = self.drop(x) + x = self.fc2(x) + x = self.drop(x) + return x + + +def window_partition(x, window_size): + """ + Args: + x: (B, H, W, C) + window_size (int): window size + Returns: + windows: (num_windows*B, window_size, window_size, C) + """ + B, H, W, C = x.shape + x = x.view(B, H // window_size, window_size, W // window_size, window_size, C) + windows = ( + x.permute(0, 1, 3, 2, 4, 5).contiguous().view(-1, window_size, window_size, C) + ) + return windows + + +def window_reverse(windows, window_size, H, W): + """ + Args: + windows: (num_windows*B, window_size, window_size, C) + window_size (int): Window size + H (int): Height of image + W (int): Width of image + Returns: + x: (B, H, W, C) + """ + B = int(windows.shape[0] / (H * W / window_size / window_size)) + x = windows.view( + B, H // window_size, W // window_size, window_size, window_size, -1 + ) + x = x.permute(0, 1, 3, 2, 4, 5).contiguous().view(B, H, W, -1) + return x + + +class WindowAttention(nn.Module): + """Window based multi-head self attention (W-MSA) module with relative position bias. + It supports both of shifted and non-shifted window. + Args: + dim (int): Number of input channels. + window_size (tuple[int]): The height and width of the window. + num_heads (int): Number of attention heads. + qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True + qk_scale (float | None, optional): Override default qk scale of head_dim ** -0.5 if set + attn_drop (float, optional): Dropout ratio of attention weight. Default: 0.0 + proj_drop (float, optional): Dropout ratio of output. Default: 0.0 + """ + + def __init__( + self, + dim, + window_size, + num_heads, + qkv_bias=True, + qk_scale=None, + attn_drop=0.0, + proj_drop=0.0, + ): + + super().__init__() + self.dim = dim + self.window_size = window_size # Wh, Ww + self.num_heads = num_heads + head_dim = dim // num_heads + self.scale = qk_scale or head_dim**-0.5 + + # define a parameter table of relative position bias + self.relative_position_bias_table = nn.Parameter( + torch.zeros((2 * window_size[0] - 1) * (2 * window_size[1] - 1), num_heads) + ) # 2*Wh-1 * 2*Ww-1, nH + + # get pair-wise relative position index for each token inside the window + coords_h = torch.arange(self.window_size[0]) + coords_w = torch.arange(self.window_size[1]) + coords = torch.stack(torch.meshgrid([coords_h, coords_w])) # 2, Wh, Ww + coords_flatten = torch.flatten(coords, 1) # 2, Wh*Ww + relative_coords = ( + coords_flatten[:, :, None] - coords_flatten[:, None, :] + ) # 2, Wh*Ww, Wh*Ww + relative_coords = relative_coords.permute( + 1, 2, 0 + ).contiguous() # Wh*Ww, Wh*Ww, 2 + relative_coords[:, :, 0] += self.window_size[0] - 1 # shift to start from 0 + relative_coords[:, :, 1] += self.window_size[1] - 1 + relative_coords[:, :, 0] *= 2 * self.window_size[1] - 1 + relative_position_index = relative_coords.sum(-1) # Wh*Ww, Wh*Ww + self.register_buffer("relative_position_index", relative_position_index) + + self.qkv = nn.Linear(dim, dim * 3, bias=qkv_bias) + self.attn_drop = nn.Dropout(attn_drop) + self.proj = nn.Linear(dim, dim) + self.proj_drop = nn.Dropout(proj_drop) + + nn.init.trunc_normal_(self.relative_position_bias_table, std=0.02) + self.softmax = nn.Softmax(dim=-1) + + def forward(self, x, mask=None): + """Forward function. + Args: + x: input features with shape of (num_windows*B, N, C) + mask: (0/-inf) mask with shape of (num_windows, Wh*Ww, Wh*Ww) or None + """ + B_, N, C = x.shape + qkv = ( + self.qkv(x) + .reshape(B_, N, 3, self.num_heads, C // self.num_heads) + .permute(2, 0, 3, 1, 4) + ) + q, k, v = ( + qkv[0], + qkv[1], + qkv[2], + ) # make torchscript happy (cannot use tensor as tuple) + + q = q * self.scale + attn = q @ k.transpose(-2, -1) + + relative_position_bias = self.relative_position_bias_table[ + self.relative_position_index.view(-1) + ].view( + self.window_size[0] * self.window_size[1], + self.window_size[0] * self.window_size[1], + -1, + ) # Wh*Ww,Wh*Ww,nH + relative_position_bias = relative_position_bias.permute( + 2, 0, 1 + ).contiguous() # nH, Wh*Ww, Wh*Ww + attn = attn + relative_position_bias.unsqueeze(0) + + if mask is not None: + nW = mask.shape[0] + attn = attn.view(B_ // nW, nW, self.num_heads, N, N) + mask.unsqueeze( + 1 + ).unsqueeze(0) + attn = attn.view(-1, self.num_heads, N, N) + attn = self.softmax(attn) + else: + attn = self.softmax(attn) + + attn = self.attn_drop(attn) + + x = (attn @ v).transpose(1, 2).reshape(B_, N, C) + x = self.proj(x) + x = self.proj_drop(x) + return x + + +class SwinTransformerBlock(nn.Module): + """Swin Transformer Block. + Args: + dim (int): Number of input channels. + num_heads (int): Number of attention heads. + window_size (int): Window size. + shift_size (int): Shift size for SW-MSA. + mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. + qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True + qk_scale (float | None, optional): Override default qk scale of head_dim ** -0.5 if set. + drop (float, optional): Dropout rate. Default: 0.0 + attn_drop (float, optional): Attention dropout rate. Default: 0.0 + drop_path (float, optional): Stochastic depth rate. Default: 0.0 + act_layer (nn.Module, optional): Activation layer. Default: nn.GELU + norm_layer (nn.Module, optional): Normalization layer. Default: nn.LayerNorm + """ + + def __init__( + self, + dim, + num_heads, + window_size=7, + shift_size=0, + mlp_ratio=4.0, + qkv_bias=True, + qk_scale=None, + drop=0.0, + attn_drop=0.0, + drop_path=0.0, + act_layer=nn.GELU, + norm_layer=nn.LayerNorm, + ): + super().__init__() + self.dim = dim + self.num_heads = num_heads + self.window_size = window_size + self.shift_size = shift_size + self.mlp_ratio = mlp_ratio + assert ( + 0 <= self.shift_size < self.window_size + ), "shift_size must in 0-window_size" + + self.norm1 = norm_layer(dim) + self.attn = WindowAttention( + dim, + window_size=to_2tuple(self.window_size), + num_heads=num_heads, + qkv_bias=qkv_bias, + qk_scale=qk_scale, + attn_drop=attn_drop, + proj_drop=drop, + ) + + self.drop_path = DropPath(drop_path) if drop_path > 0.0 else nn.Identity() + self.norm2 = norm_layer(dim) + mlp_hidden_dim = int(dim * mlp_ratio) + self.mlp = Mlp( + in_features=dim, + hidden_features=mlp_hidden_dim, + act_layer=act_layer, + drop=drop, + ) + + self.H = None + self.W = None + + def forward(self, x, mask_matrix): + """Forward function. + Args: + x: Input feature, tensor size (B, H*W, C). + H, W: Spatial resolution of the input feature. + mask_matrix: Attention mask for cyclic shift. + """ + B, L, C = x.shape + H, W = self.H, self.W + assert L == H * W, "input feature has wrong size" + + shortcut = x + x = self.norm1(x) + x = x.view(B, H, W, C) + + # pad feature maps to multiples of window size + pad_l = pad_t = 0 + pad_r = (self.window_size - W % self.window_size) % self.window_size + pad_b = (self.window_size - H % self.window_size) % self.window_size + x = F.pad(x, (0, 0, pad_l, pad_r, pad_t, pad_b)) + _, Hp, Wp, _ = x.shape + + # cyclic shift + if self.shift_size > 0: + shifted_x = torch.roll( + x, shifts=(-self.shift_size, -self.shift_size), dims=(1, 2) + ) + attn_mask = mask_matrix + else: + shifted_x = x + attn_mask = None + + # partition windows + x_windows = window_partition( + shifted_x, self.window_size + ) # nW*B, window_size, window_size, C + x_windows = x_windows.view( + -1, self.window_size * self.window_size, C + ) # nW*B, window_size*window_size, C + + # W-MSA/SW-MSA + attn_windows = self.attn( + x_windows, mask=attn_mask + ) # nW*B, window_size*window_size, C + + # merge windows + attn_windows = attn_windows.view(-1, self.window_size, self.window_size, C) + shifted_x = window_reverse(attn_windows, self.window_size, Hp, Wp) # B H' W' C + + # reverse cyclic shift + if self.shift_size > 0: + x = torch.roll( + shifted_x, shifts=(self.shift_size, self.shift_size), dims=(1, 2) + ) + else: + x = shifted_x + + if pad_r > 0 or pad_b > 0: + x = x[:, :H, :W, :].contiguous() + + x = x.view(B, H * W, C) + + # FFN + x = shortcut + self.drop_path(x) + x = x + self.drop_path(self.mlp(self.norm2(x))) + + return x + + +class PatchMerging(nn.Module): + """Patch Merging Layer + Args: + dim (int): Number of input channels. + norm_layer (nn.Module, optional): Normalization layer. Default: nn.LayerNorm + """ + + def __init__(self, dim, norm_layer=nn.LayerNorm): + super().__init__() + self.dim = dim + self.reduction = nn.Linear(4 * dim, 2 * dim, bias=False) + self.norm = norm_layer(4 * dim) + + def forward(self, x, H, W): + """Forward function. + Args: + x: Input feature, tensor size (B, H*W, C). + H, W: Spatial resolution of the input feature. + """ + B, L, C = x.shape + assert L == H * W, "input feature has wrong size" + + x = x.view(B, H, W, C) + + # padding + pad_input = (H % 2 == 1) or (W % 2 == 1) + if pad_input: + x = F.pad(x, (0, 0, 0, W % 2, 0, H % 2)) + + x0 = x[:, 0::2, 0::2, :] # B H/2 W/2 C + x1 = x[:, 1::2, 0::2, :] # B H/2 W/2 C + x2 = x[:, 0::2, 1::2, :] # B H/2 W/2 C + x3 = x[:, 1::2, 1::2, :] # B H/2 W/2 C + x = torch.cat([x0, x1, x2, x3], -1) # B H/2 W/2 4*C + x = x.view(B, -1, 4 * C) # B H/2*W/2 4*C + + x = self.norm(x) + x = self.reduction(x) + + return x + + +class BasicLayer(nn.Module): + """A basic Swin Transformer layer for one stage. + Args: + dim (int): Number of feature channels + depth (int): Depths of this stage. + num_heads (int): Number of attention head. + window_size (int): Local window size. Default: 7. + mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. Default: 4. + qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True + qk_scale (float | None, optional): Override default qk scale of head_dim ** -0.5 if set. + drop (float, optional): Dropout rate. Default: 0.0 + attn_drop (float, optional): Attention dropout rate. Default: 0.0 + drop_path (float | tuple[float], optional): Stochastic depth rate. Default: 0.0 + norm_layer (nn.Module, optional): Normalization layer. Default: nn.LayerNorm + downsample (nn.Module | None, optional): Downsample layer at the end of the layer. Default: None + use_checkpoint (bool): Whether to use checkpointing to save memory. Default: False. + """ + + def __init__( + self, + dim, + depth, + num_heads, + window_size=7, + mlp_ratio=4.0, + qkv_bias=True, + qk_scale=None, + drop=0.0, + attn_drop=0.0, + drop_path=0.0, + norm_layer=nn.LayerNorm, + downsample=None, + use_checkpoint=False, + ): + super().__init__() + self.window_size = window_size + self.shift_size = window_size // 2 + self.depth = depth + self.use_checkpoint = use_checkpoint + + # build blocks + self.blocks = nn.ModuleList( + [ + SwinTransformerBlock( + dim=dim, + num_heads=num_heads, + window_size=window_size, + shift_size=0 if (i % 2 == 0) else window_size // 2, + mlp_ratio=mlp_ratio, + qkv_bias=qkv_bias, + qk_scale=qk_scale, + drop=drop, + attn_drop=attn_drop, + drop_path=( + drop_path[i] if isinstance(drop_path, list) else drop_path + ), + norm_layer=norm_layer, + ) + for i in range(depth) + ] + ) + + # patch merging layer + if downsample is not None: + self.downsample = downsample(dim=dim, norm_layer=norm_layer) + else: + self.downsample = None + + def forward(self, x, H, W): + """Forward function. + Args: + x: Input feature, tensor size (B, H*W, C). + H, W: Spatial resolution of the input feature. + """ + + # calculate attention mask for SW-MSA + Hp = int(np.ceil(H / self.window_size)) * self.window_size + Wp = int(np.ceil(W / self.window_size)) * self.window_size + img_mask = torch.zeros((1, Hp, Wp, 1), device=x.device) # 1 Hp Wp 1 + h_slices = ( + slice(0, -self.window_size), + slice(-self.window_size, -self.shift_size), + slice(-self.shift_size, None), + ) + w_slices = ( + slice(0, -self.window_size), + slice(-self.window_size, -self.shift_size), + slice(-self.shift_size, None), + ) + cnt = 0 + for h in h_slices: + for w in w_slices: + img_mask[:, h, w, :] = cnt + cnt += 1 + + mask_windows = window_partition( + img_mask, self.window_size + ) # nW, window_size, window_size, 1 + mask_windows = mask_windows.view(-1, self.window_size * self.window_size) + attn_mask = mask_windows.unsqueeze(1) - mask_windows.unsqueeze(2) + attn_mask = attn_mask.masked_fill(attn_mask != 0, float(-100.0)).masked_fill( + attn_mask == 0, float(0.0) + ) + + for blk in self.blocks: + blk.H, blk.W = H, W + if self.use_checkpoint: + x = checkpoint.checkpoint(blk, x, attn_mask) + else: + x = blk(x, attn_mask) + if self.downsample is not None: + x_down = self.downsample(x, H, W) + Wh, Ww = (H + 1) // 2, (W + 1) // 2 + return x, H, W, x_down, Wh, Ww + else: + return x, H, W, x, H, W + + +class PatchEmbed(nn.Module): + """Image to Patch Embedding + Args: + patch_size (int): Patch token size. Default: 4. + in_chans (int): Number of input image channels. Default: 3. + embed_dim (int): Number of linear projection output channels. Default: 96. + norm_layer (nn.Module, optional): Normalization layer. Default: None + """ + + def __init__(self, patch_size=4, in_chans=3, embed_dim=96, norm_layer=None): + super().__init__() + patch_size = to_2tuple(patch_size) + self.patch_size = patch_size + + self.in_chans = in_chans + self.embed_dim = embed_dim + + self.proj = nn.Conv2d( + in_chans, embed_dim, kernel_size=patch_size, stride=patch_size + ) + if norm_layer is not None: + self.norm = norm_layer(embed_dim) + else: + self.norm = None + + def forward(self, x): + """Forward function.""" + # padding + _, _, H, W = x.size() + if W % self.patch_size[1] != 0: + x = F.pad(x, (0, self.patch_size[1] - W % self.patch_size[1])) + if H % self.patch_size[0] != 0: + x = F.pad(x, (0, 0, 0, self.patch_size[0] - H % self.patch_size[0])) + + x = self.proj(x) # B C Wh Ww + if self.norm is not None: + Wh, Ww = x.size(2), x.size(3) + x = x.flatten(2).transpose(1, 2) + x = self.norm(x) + x = x.transpose(1, 2).view(-1, self.embed_dim, Wh, Ww) + + return x + + +class SwinTransformer(nn.Module): + """Swin Transformer backbone. + A PyTorch impl of : `Swin Transformer: Hierarchical Vision Transformer using Shifted Windows` - + https://arxiv.org/pdf/2103.14030 + Args: + pretrain_img_size (int): Input image size for training the pretrained model, + used in absolute postion embedding. Default 224. + patch_size (int | tuple(int)): Patch size. Default: 4. + in_chans (int): Number of input image channels. Default: 3. + embed_dim (int): Number of linear projection output channels. Default: 96. + depths (tuple[int]): Depths of each Swin Transformer stage. + num_heads (tuple[int]): Number of attention head of each stage. + window_size (int): Window size. Default: 7. + mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. Default: 4. + qkv_bias (bool): If True, add a learnable bias to query, key, value. Default: True + qk_scale (float): Override default qk scale of head_dim ** -0.5 if set. + drop_rate (float): Dropout rate. + attn_drop_rate (float): Attention dropout rate. Default: 0. + drop_path_rate (float): Stochastic depth rate. Default: 0.2. + norm_layer (nn.Module): Normalization layer. Default: nn.LayerNorm. + ape (bool): If True, add absolute position embedding to the patch embedding. Default: False. + patch_norm (bool): If True, add normalization after patch embedding. Default: True. + out_indices (Sequence[int]): Output from which stages. + frozen_stages (int): Stages to be frozen (stop grad and set eval mode). + -1 means not freezing any parameters. + use_checkpoint (bool): Whether to use checkpointing to save memory. Default: False. + """ + + def __init__( + self, + pretrain_img_size=224, + patch_size=4, + in_chans=3, + embed_dim=96, + depths=[2, 2, 6, 2], + num_heads=[3, 6, 12, 24], + window_size=7, + mlp_ratio=4.0, + qkv_bias=True, + qk_scale=None, + drop_rate=0.0, + attn_drop_rate=0.0, + drop_path_rate=0.2, + norm_layer=nn.LayerNorm, + ape=False, + patch_norm=True, + out_indices=(0, 1, 2, 3), + frozen_stages=-1, + use_checkpoint=False, + ): + super().__init__() + + self.pretrain_img_size = pretrain_img_size + self.num_layers = len(depths) + self.embed_dim = embed_dim + self.ape = ape + self.patch_norm = patch_norm + self.out_indices = out_indices + self.frozen_stages = frozen_stages + + # split image into non-overlapping patches + self.patch_embed = PatchEmbed( + patch_size=patch_size, + in_chans=in_chans, + embed_dim=embed_dim, + norm_layer=norm_layer if self.patch_norm else None, + ) + + # absolute position embedding + if self.ape: + pretrain_img_size = to_2tuple(pretrain_img_size) + patch_size = to_2tuple(patch_size) + patches_resolution = [ + pretrain_img_size[0] // patch_size[0], + pretrain_img_size[1] // patch_size[1], + ] + + self.absolute_pos_embed = nn.Parameter( + torch.zeros(1, embed_dim, patches_resolution[0], patches_resolution[1]) + ) + nn.init.trunc_normal_(self.absolute_pos_embed, std=0.02) + + self.pos_drop = nn.Dropout(p=drop_rate) + + # stochastic depth + dpr = [ + x.item() for x in torch.linspace(0, drop_path_rate, sum(depths)) + ] # stochastic depth decay rule + + # build layers + self.layers = nn.ModuleList() + for i_layer in range(self.num_layers): + layer = BasicLayer( + dim=int(embed_dim * 2**i_layer), + depth=depths[i_layer], + num_heads=num_heads[i_layer], + window_size=window_size, + mlp_ratio=mlp_ratio, + qkv_bias=qkv_bias, + qk_scale=qk_scale, + drop=drop_rate, + attn_drop=attn_drop_rate, + drop_path=dpr[sum(depths[:i_layer]) : sum(depths[: i_layer + 1])], + norm_layer=norm_layer, + downsample=PatchMerging if (i_layer < self.num_layers - 1) else None, + use_checkpoint=use_checkpoint, + ) + self.layers.append(layer) + + num_features = [int(embed_dim * 2**i) for i in range(self.num_layers)] + self.num_features = num_features + + # add a norm layer for each output + for i_layer in out_indices: + layer = norm_layer(num_features[i_layer]) + layer_name = f"norm{i_layer}" + self.add_module(layer_name, layer) + + self._freeze_stages() + + def _freeze_stages(self): + if self.frozen_stages >= 0: + self.patch_embed.eval() + for param in self.patch_embed.parameters(): + param.requires_grad = False + + if self.frozen_stages >= 1 and self.ape: + self.absolute_pos_embed.requires_grad = False + + if self.frozen_stages >= 2: + self.pos_drop.eval() + for i in range(0, self.frozen_stages - 1): + m = self.layers[i] + m.eval() + for param in m.parameters(): + param.requires_grad = False + + def init_weights(self, pretrained=None): + """Initialize the weights in backbone. + Args: + pretrained (str, optional): Path to pre-trained weights. + Defaults to None. + """ + + def _init_weights(m): + if isinstance(m, nn.Linear): + nn.init.trunc_normal_(m.weight, std=0.02) + if isinstance(m, nn.Linear) and m.bias is not None: + nn.init.constant_(m.bias, 0) + elif isinstance(m, nn.LayerNorm): + nn.init.constant_(m.bias, 0) + nn.init.constant_(m.weight, 1.0) + + def forward(self, x): + """Forward function.""" + x = self.patch_embed(x) + + Wh, Ww = x.size(2), x.size(3) + if self.ape: + # interpolate the position embedding to the corresponding size + absolute_pos_embed = F.interpolate( + self.absolute_pos_embed, size=(Wh, Ww), mode="bicubic" + ) + x = (x + absolute_pos_embed).flatten(2).transpose(1, 2) # B Wh*Ww C + else: + x = x.flatten(2).transpose(1, 2) + x = self.pos_drop(x) + + outs = {} + for i in range(self.num_layers): + layer = self.layers[i] + x_out, H, W, x, Wh, Ww = layer(x, Wh, Ww) + + if i in self.out_indices: + norm_layer = getattr(self, f"norm{i}") + x_out = norm_layer(x_out) + + out = ( + x_out.view(-1, H, W, self.num_features[i]) + .permute(0, 3, 1, 2) + .contiguous() + ) + outs["res{}".format(i + 2)] = out + + return outs + + def train(self, mode=True): + """Convert the model into training mode while keep layers freezed.""" + super(SwinTransformer, self).train(mode) + self._freeze_stages() diff --git a/perception_models/apps/detection/DETA_pe/models/utils_d2.py b/perception_models/apps/detection/DETA_pe/models/utils_d2.py new file mode 100644 index 0000000000000000000000000000000000000000..2b89a4c3fbe079a77fd0cef947cf9ada787fc55d --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/models/utils_d2.py @@ -0,0 +1,186 @@ +# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved +import math +import torch +import torch.nn as nn +import torch.nn.functional as F + +__all__ = [ + "window_partition", + "window_unpartition", + "add_decomposed_rel_pos", + "get_abs_pos", + "PatchEmbed", +] + + +def window_partition(x, window_size): + """ + Partition into non-overlapping windows with padding if needed. + Args: + x (tensor): input tokens with [B, H, W, C]. + window_size (int): window size. + + Returns: + windows: windows after partition with [B * num_windows, window_size, window_size, C]. + (Hp, Wp): padded height and width before partition + """ + B, H, W, C = x.shape + + pad_h = (window_size - H % window_size) % window_size + pad_w = (window_size - W % window_size) % window_size + if pad_h > 0 or pad_w > 0: + x = F.pad(x, (0, 0, 0, pad_w, 0, pad_h)) + Hp, Wp = H + pad_h, W + pad_w + + x = x.view(B, Hp // window_size, window_size, Wp // window_size, window_size, C) + windows = x.permute(0, 1, 3, 2, 4, 5).contiguous().view(-1, window_size, window_size, C) + return windows, (Hp, Wp) + + +def window_unpartition(windows, window_size, pad_hw, hw): + """ + Window unpartition into original sequences and removing padding. + Args: + x (tensor): input tokens with [B * num_windows, window_size, window_size, C]. + window_size (int): window size. + pad_hw (Tuple): padded height and width (Hp, Wp). + hw (Tuple): original height and width (H, W) before padding. + + Returns: + x: unpartitioned sequences with [B, H, W, C]. + """ + Hp, Wp = pad_hw + H, W = hw + B = windows.shape[0] // (Hp * Wp // window_size // window_size) + x = windows.view(B, Hp // window_size, Wp // window_size, window_size, window_size, -1) + x = x.permute(0, 1, 3, 2, 4, 5).contiguous().view(B, Hp, Wp, -1) + + if Hp > H or Wp > W: + x = x[:, :H, :W, :].contiguous() + return x + + +def get_rel_pos(q_size, k_size, rel_pos): + """ + Get relative positional embeddings according to the relative positions of + query and key sizes. + Args: + q_size (int): size of query q. + k_size (int): size of key k. + rel_pos (Tensor): relative position embeddings (L, C). + + Returns: + Extracted positional embeddings according to relative positions. + """ + max_rel_dist = int(2 * max(q_size, k_size) - 1) + # Interpolate rel pos if needed. + if rel_pos.shape[0] != max_rel_dist: + # Interpolate rel pos. + rel_pos_resized = F.interpolate( + rel_pos.reshape(1, rel_pos.shape[0], -1).permute(0, 2, 1), + size=max_rel_dist, + mode="linear", + ) + rel_pos_resized = rel_pos_resized.reshape(-1, max_rel_dist).permute(1, 0) + else: + rel_pos_resized = rel_pos + + # Scale the coords with short length if shapes for q and k are different. + q_coords = torch.arange(q_size)[:, None] * max(k_size / q_size, 1.0) + k_coords = torch.arange(k_size)[None, :] * max(q_size / k_size, 1.0) + relative_coords = (q_coords - k_coords) + (k_size - 1) * max(q_size / k_size, 1.0) + + return rel_pos_resized[relative_coords.long()] + + +def add_decomposed_rel_pos(attn, q, rel_pos_h, rel_pos_w, q_size, k_size): + """ + Calculate decomposed Relative Positional Embeddings from :paper:`mvitv2`. + https://github.com/facebookresearch/mvit/blob/19786631e330df9f3622e5402b4a419a263a2c80/mvit/models/attention.py # noqa B950 + Args: + attn (Tensor): attention map. + q (Tensor): query q in the attention layer with shape (B, q_h * q_w, C). + rel_pos_h (Tensor): relative position embeddings (Lh, C) for height axis. + rel_pos_w (Tensor): relative position embeddings (Lw, C) for width axis. + q_size (Tuple): spatial sequence size of query q with (q_h, q_w). + k_size (Tuple): spatial sequence size of key k with (k_h, k_w). + + Returns: + attn (Tensor): attention map with added relative positional embeddings. + """ + q_h, q_w = q_size + k_h, k_w = k_size + Rh = get_rel_pos(q_h, k_h, rel_pos_h) + Rw = get_rel_pos(q_w, k_w, rel_pos_w) + + B, _, dim = q.shape + r_q = q.reshape(B, q_h, q_w, dim) + rel_h = torch.einsum("bhwc,hkc->bhwk", r_q, Rh) + rel_w = torch.einsum("bhwc,wkc->bhwk", r_q, Rw) + + attn = ( + attn.view(B, q_h, q_w, k_h, k_w) + rel_h[:, :, :, :, None] + rel_w[:, :, :, None, :] + ).view(B, q_h * q_w, k_h * k_w) + + return attn + + +def get_abs_pos(abs_pos, has_cls_token, hw): + """ + Calculate absolute positional embeddings. If needed, resize embeddings and remove cls_token + dimension for the original embeddings. + Args: + abs_pos (Tensor): absolute positional embeddings with (1, num_position, C). + has_cls_token (bool): If true, has 1 embedding in abs_pos for cls token. + hw (Tuple): size of input image tokens. + + Returns: + Absolute positional embeddings after processing with shape (1, H, W, C) + """ + h, w = hw + if has_cls_token: + abs_pos = abs_pos[:, 1:] + xy_num = abs_pos.shape[1] + size = int(math.sqrt(xy_num)) + assert size * size == xy_num + + if size != h or size != w: + new_abs_pos = F.interpolate( + abs_pos.reshape(1, size, size, -1).permute(0, 3, 1, 2), + size=(h, w), + mode="bicubic", + align_corners=False, + ) + + return new_abs_pos.permute(0, 2, 3, 1) + else: + return abs_pos.reshape(1, h, w, -1) + + +class PatchEmbed(nn.Module): + """ + Image to Patch Embedding. + """ + + def __init__( + self, kernel_size=(16, 16), stride=(16, 16), padding=(0, 0), in_chans=3, embed_dim=768 + ): + """ + Args: + kernel_size (Tuple): kernel size of the projection layer. + stride (Tuple): stride of the projection layer. + padding (Tuple): padding size of the projection layer. + in_chans (int): Number of input image channels. + embed_dim (int): embed_dim (int): Patch embedding dimension. + """ + super().__init__() + + self.proj = nn.Conv2d( + in_chans, embed_dim, kernel_size=kernel_size, stride=stride, padding=padding + ) + + def forward(self, x): + x = self.proj(x) + # B C H W -> B H W C + x = x.permute(0, 2, 3, 1) + return x diff --git a/perception_models/apps/detection/DETA_pe/models/utils_fed_loss.py b/perception_models/apps/detection/DETA_pe/models/utils_fed_loss.py new file mode 100644 index 0000000000000000000000000000000000000000..bafc64d128403a749dae5fa99af4f3f8e3535ade --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/models/utils_fed_loss.py @@ -0,0 +1,38 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +import json + +import numpy as np +import torch +from torch.nn import functional as F + + + +def load_class_freq( + path="datasets/metadata/lvis_v1_train_cat_info.json", freq_weight=1.0 +): + # cat_info = json.load(open(path, "r")) + cat_info = LVIS_CATEGORY_IMAGE_COUNT + cat_info = torch.tensor( + [c["image_count"] for c in sorted(cat_info, key=lambda x: x["id"])] + ) + freq_weight = cat_info.float() ** freq_weight + return freq_weight + + +def get_fed_loss_inds(gt_classes, num_sample_cats, C, weight=None): + appeared = torch.unique(gt_classes) # C' + prob = appeared.new_ones(C + 1).float() + prob[-1] = 0 + if len(appeared) < num_sample_cats: + if weight is not None: + prob[:C] = weight.float().clone() + prob[appeared] = 0 + more_appeared = torch.multinomial( + prob, num_sample_cats - len(appeared), replacement=False + ) + appeared = torch.cat([appeared, more_appeared]) + return appeared + + +# fmt: off +LVIS_CATEGORY_IMAGE_COUNT = [{'id': 1, 'image_count': 64}, {'id': 2, 'image_count': 364}, {'id': 3, 'image_count': 1911}, {'id': 4, 'image_count': 149}, {'id': 5, 'image_count': 29}, {'id': 6, 'image_count': 26}, {'id': 7, 'image_count': 59}, {'id': 8, 'image_count': 22}, {'id': 9, 'image_count': 12}, {'id': 10, 'image_count': 28}, {'id': 11, 'image_count': 505}, {'id': 12, 'image_count': 1207}, {'id': 13, 'image_count': 4}, {'id': 14, 'image_count': 10}, {'id': 15, 'image_count': 500}, {'id': 16, 'image_count': 33}, {'id': 17, 'image_count': 3}, {'id': 18, 'image_count': 44}, {'id': 19, 'image_count': 561}, {'id': 20, 'image_count': 8}, {'id': 21, 'image_count': 9}, {'id': 22, 'image_count': 33}, {'id': 23, 'image_count': 1883}, {'id': 24, 'image_count': 98}, {'id': 25, 'image_count': 70}, {'id': 26, 'image_count': 46}, {'id': 27, 'image_count': 117}, {'id': 28, 'image_count': 41}, {'id': 29, 'image_count': 1395}, {'id': 30, 'image_count': 7}, {'id': 31, 'image_count': 1}, {'id': 32, 'image_count': 314}, {'id': 33, 'image_count': 31}, {'id': 34, 'image_count': 1905}, {'id': 35, 'image_count': 1859}, {'id': 36, 'image_count': 1623}, {'id': 37, 'image_count': 47}, {'id': 38, 'image_count': 3}, {'id': 39, 'image_count': 3}, {'id': 40, 'image_count': 1}, {'id': 41, 'image_count': 305}, {'id': 42, 'image_count': 6}, {'id': 43, 'image_count': 210}, {'id': 44, 'image_count': 36}, {'id': 45, 'image_count': 1787}, {'id': 46, 'image_count': 17}, {'id': 47, 'image_count': 51}, {'id': 48, 'image_count': 138}, {'id': 49, 'image_count': 3}, {'id': 50, 'image_count': 1470}, {'id': 51, 'image_count': 3}, {'id': 52, 'image_count': 2}, {'id': 53, 'image_count': 186}, {'id': 54, 'image_count': 76}, {'id': 55, 'image_count': 26}, {'id': 56, 'image_count': 303}, {'id': 57, 'image_count': 738}, {'id': 58, 'image_count': 1799}, {'id': 59, 'image_count': 1934}, {'id': 60, 'image_count': 1609}, {'id': 61, 'image_count': 1622}, {'id': 62, 'image_count': 41}, {'id': 63, 'image_count': 4}, {'id': 64, 'image_count': 11}, {'id': 65, 'image_count': 270}, {'id': 66, 'image_count': 349}, {'id': 67, 'image_count': 42}, {'id': 68, 'image_count': 823}, {'id': 69, 'image_count': 6}, {'id': 70, 'image_count': 48}, {'id': 71, 'image_count': 3}, {'id': 72, 'image_count': 42}, {'id': 73, 'image_count': 24}, {'id': 74, 'image_count': 16}, {'id': 75, 'image_count': 605}, {'id': 76, 'image_count': 646}, {'id': 77, 'image_count': 1765}, {'id': 78, 'image_count': 2}, {'id': 79, 'image_count': 125}, {'id': 80, 'image_count': 1420}, {'id': 81, 'image_count': 140}, {'id': 82, 'image_count': 4}, {'id': 83, 'image_count': 322}, {'id': 84, 'image_count': 60}, {'id': 85, 'image_count': 2}, {'id': 86, 'image_count': 231}, {'id': 87, 'image_count': 333}, {'id': 88, 'image_count': 1941}, {'id': 89, 'image_count': 367}, {'id': 90, 'image_count': 1922}, {'id': 91, 'image_count': 18}, {'id': 92, 'image_count': 81}, {'id': 93, 'image_count': 1}, {'id': 94, 'image_count': 1852}, {'id': 95, 'image_count': 430}, {'id': 96, 'image_count': 247}, {'id': 97, 'image_count': 94}, {'id': 98, 'image_count': 21}, {'id': 99, 'image_count': 1821}, {'id': 100, 'image_count': 16}, {'id': 101, 'image_count': 12}, {'id': 102, 'image_count': 25}, {'id': 103, 'image_count': 41}, {'id': 104, 'image_count': 244}, {'id': 105, 'image_count': 7}, {'id': 106, 'image_count': 1}, {'id': 107, 'image_count': 40}, {'id': 108, 'image_count': 40}, {'id': 109, 'image_count': 104}, {'id': 110, 'image_count': 1671}, {'id': 111, 'image_count': 49}, {'id': 112, 'image_count': 243}, {'id': 113, 'image_count': 2}, {'id': 114, 'image_count': 242}, {'id': 115, 'image_count': 271}, {'id': 116, 'image_count': 104}, {'id': 117, 'image_count': 8}, {'id': 118, 'image_count': 1758}, {'id': 119, 'image_count': 1}, {'id': 120, 'image_count': 48}, {'id': 121, 'image_count': 14}, {'id': 122, 'image_count': 40}, {'id': 123, 'image_count': 1}, {'id': 124, 'image_count': 37}, {'id': 125, 'image_count': 1510}, {'id': 126, 'image_count': 6}, {'id': 127, 'image_count': 1903}, {'id': 128, 'image_count': 70}, {'id': 129, 'image_count': 86}, {'id': 130, 'image_count': 7}, {'id': 131, 'image_count': 5}, {'id': 132, 'image_count': 1406}, {'id': 133, 'image_count': 1901}, {'id': 134, 'image_count': 15}, {'id': 135, 'image_count': 28}, {'id': 136, 'image_count': 6}, {'id': 137, 'image_count': 494}, {'id': 138, 'image_count': 234}, {'id': 139, 'image_count': 1922}, {'id': 140, 'image_count': 1}, {'id': 141, 'image_count': 35}, {'id': 142, 'image_count': 5}, {'id': 143, 'image_count': 1828}, {'id': 144, 'image_count': 8}, {'id': 145, 'image_count': 63}, {'id': 146, 'image_count': 1668}, {'id': 147, 'image_count': 4}, {'id': 148, 'image_count': 95}, {'id': 149, 'image_count': 17}, {'id': 150, 'image_count': 1567}, {'id': 151, 'image_count': 2}, {'id': 152, 'image_count': 103}, {'id': 153, 'image_count': 50}, {'id': 154, 'image_count': 1309}, {'id': 155, 'image_count': 6}, {'id': 156, 'image_count': 92}, {'id': 157, 'image_count': 19}, {'id': 158, 'image_count': 37}, {'id': 159, 'image_count': 4}, {'id': 160, 'image_count': 709}, {'id': 161, 'image_count': 9}, {'id': 162, 'image_count': 82}, {'id': 163, 'image_count': 15}, {'id': 164, 'image_count': 3}, {'id': 165, 'image_count': 61}, {'id': 166, 'image_count': 51}, {'id': 167, 'image_count': 5}, {'id': 168, 'image_count': 13}, {'id': 169, 'image_count': 642}, {'id': 170, 'image_count': 24}, {'id': 171, 'image_count': 255}, {'id': 172, 'image_count': 9}, {'id': 173, 'image_count': 1808}, {'id': 174, 'image_count': 31}, {'id': 175, 'image_count': 158}, {'id': 176, 'image_count': 80}, {'id': 177, 'image_count': 1884}, {'id': 178, 'image_count': 158}, {'id': 179, 'image_count': 2}, {'id': 180, 'image_count': 12}, {'id': 181, 'image_count': 1659}, {'id': 182, 'image_count': 7}, {'id': 183, 'image_count': 834}, {'id': 184, 'image_count': 57}, {'id': 185, 'image_count': 174}, {'id': 186, 'image_count': 95}, {'id': 187, 'image_count': 27}, {'id': 188, 'image_count': 22}, {'id': 189, 'image_count': 1391}, {'id': 190, 'image_count': 90}, {'id': 191, 'image_count': 40}, {'id': 192, 'image_count': 445}, {'id': 193, 'image_count': 21}, {'id': 194, 'image_count': 1132}, {'id': 195, 'image_count': 177}, {'id': 196, 'image_count': 4}, {'id': 197, 'image_count': 17}, {'id': 198, 'image_count': 84}, {'id': 199, 'image_count': 55}, {'id': 200, 'image_count': 30}, {'id': 201, 'image_count': 25}, {'id': 202, 'image_count': 2}, {'id': 203, 'image_count': 125}, {'id': 204, 'image_count': 1135}, {'id': 205, 'image_count': 19}, {'id': 206, 'image_count': 72}, {'id': 207, 'image_count': 1926}, {'id': 208, 'image_count': 159}, {'id': 209, 'image_count': 7}, {'id': 210, 'image_count': 1}, {'id': 211, 'image_count': 13}, {'id': 212, 'image_count': 35}, {'id': 213, 'image_count': 18}, {'id': 214, 'image_count': 8}, {'id': 215, 'image_count': 6}, {'id': 216, 'image_count': 35}, {'id': 217, 'image_count': 1222}, {'id': 218, 'image_count': 103}, {'id': 219, 'image_count': 28}, {'id': 220, 'image_count': 63}, {'id': 221, 'image_count': 28}, {'id': 222, 'image_count': 5}, {'id': 223, 'image_count': 7}, {'id': 224, 'image_count': 14}, {'id': 225, 'image_count': 1918}, {'id': 226, 'image_count': 133}, {'id': 227, 'image_count': 16}, {'id': 228, 'image_count': 27}, {'id': 229, 'image_count': 110}, {'id': 230, 'image_count': 1895}, {'id': 231, 'image_count': 4}, {'id': 232, 'image_count': 1927}, {'id': 233, 'image_count': 8}, {'id': 234, 'image_count': 1}, {'id': 235, 'image_count': 263}, {'id': 236, 'image_count': 10}, {'id': 237, 'image_count': 2}, {'id': 238, 'image_count': 3}, {'id': 239, 'image_count': 87}, {'id': 240, 'image_count': 9}, {'id': 241, 'image_count': 71}, {'id': 242, 'image_count': 13}, {'id': 243, 'image_count': 18}, {'id': 244, 'image_count': 2}, {'id': 245, 'image_count': 5}, {'id': 246, 'image_count': 45}, {'id': 247, 'image_count': 1}, {'id': 248, 'image_count': 23}, {'id': 249, 'image_count': 32}, {'id': 250, 'image_count': 4}, {'id': 251, 'image_count': 1}, {'id': 252, 'image_count': 858}, {'id': 253, 'image_count': 661}, {'id': 254, 'image_count': 168}, {'id': 255, 'image_count': 210}, {'id': 256, 'image_count': 65}, {'id': 257, 'image_count': 4}, {'id': 258, 'image_count': 2}, {'id': 259, 'image_count': 159}, {'id': 260, 'image_count': 31}, {'id': 261, 'image_count': 811}, {'id': 262, 'image_count': 1}, {'id': 263, 'image_count': 42}, {'id': 264, 'image_count': 27}, {'id': 265, 'image_count': 2}, {'id': 266, 'image_count': 5}, {'id': 267, 'image_count': 95}, {'id': 268, 'image_count': 32}, {'id': 269, 'image_count': 1}, {'id': 270, 'image_count': 1}, {'id': 271, 'image_count': 1844}, {'id': 272, 'image_count': 897}, {'id': 273, 'image_count': 31}, {'id': 274, 'image_count': 23}, {'id': 275, 'image_count': 1}, {'id': 276, 'image_count': 202}, {'id': 277, 'image_count': 746}, {'id': 278, 'image_count': 44}, {'id': 279, 'image_count': 14}, {'id': 280, 'image_count': 26}, {'id': 281, 'image_count': 1}, {'id': 282, 'image_count': 2}, {'id': 283, 'image_count': 25}, {'id': 284, 'image_count': 238}, {'id': 285, 'image_count': 592}, {'id': 286, 'image_count': 26}, {'id': 287, 'image_count': 5}, {'id': 288, 'image_count': 42}, {'id': 289, 'image_count': 13}, {'id': 290, 'image_count': 46}, {'id': 291, 'image_count': 1}, {'id': 292, 'image_count': 8}, {'id': 293, 'image_count': 34}, {'id': 294, 'image_count': 5}, {'id': 295, 'image_count': 1}, {'id': 296, 'image_count': 1871}, {'id': 297, 'image_count': 717}, {'id': 298, 'image_count': 1010}, {'id': 299, 'image_count': 679}, {'id': 300, 'image_count': 3}, {'id': 301, 'image_count': 4}, {'id': 302, 'image_count': 1}, {'id': 303, 'image_count': 166}, {'id': 304, 'image_count': 2}, {'id': 305, 'image_count': 266}, {'id': 306, 'image_count': 101}, {'id': 307, 'image_count': 6}, {'id': 308, 'image_count': 14}, {'id': 309, 'image_count': 133}, {'id': 310, 'image_count': 2}, {'id': 311, 'image_count': 38}, {'id': 312, 'image_count': 95}, {'id': 313, 'image_count': 1}, {'id': 314, 'image_count': 12}, {'id': 315, 'image_count': 49}, {'id': 316, 'image_count': 5}, {'id': 317, 'image_count': 5}, {'id': 318, 'image_count': 16}, {'id': 319, 'image_count': 216}, {'id': 320, 'image_count': 12}, {'id': 321, 'image_count': 1}, {'id': 322, 'image_count': 54}, {'id': 323, 'image_count': 5}, {'id': 324, 'image_count': 245}, {'id': 325, 'image_count': 12}, {'id': 326, 'image_count': 7}, {'id': 327, 'image_count': 35}, {'id': 328, 'image_count': 36}, {'id': 329, 'image_count': 32}, {'id': 330, 'image_count': 1027}, {'id': 331, 'image_count': 10}, {'id': 332, 'image_count': 12}, {'id': 333, 'image_count': 1}, {'id': 334, 'image_count': 67}, {'id': 335, 'image_count': 71}, {'id': 336, 'image_count': 30}, {'id': 337, 'image_count': 48}, {'id': 338, 'image_count': 249}, {'id': 339, 'image_count': 13}, {'id': 340, 'image_count': 29}, {'id': 341, 'image_count': 14}, {'id': 342, 'image_count': 236}, {'id': 343, 'image_count': 15}, {'id': 344, 'image_count': 1521}, {'id': 345, 'image_count': 25}, {'id': 346, 'image_count': 249}, {'id': 347, 'image_count': 139}, {'id': 348, 'image_count': 2}, {'id': 349, 'image_count': 2}, {'id': 350, 'image_count': 1890}, {'id': 351, 'image_count': 1240}, {'id': 352, 'image_count': 1}, {'id': 353, 'image_count': 9}, {'id': 354, 'image_count': 1}, {'id': 355, 'image_count': 3}, {'id': 356, 'image_count': 11}, {'id': 357, 'image_count': 4}, {'id': 358, 'image_count': 236}, {'id': 359, 'image_count': 44}, {'id': 360, 'image_count': 19}, {'id': 361, 'image_count': 1100}, {'id': 362, 'image_count': 7}, {'id': 363, 'image_count': 69}, {'id': 364, 'image_count': 2}, {'id': 365, 'image_count': 8}, {'id': 366, 'image_count': 5}, {'id': 367, 'image_count': 227}, {'id': 368, 'image_count': 6}, {'id': 369, 'image_count': 106}, {'id': 370, 'image_count': 81}, {'id': 371, 'image_count': 17}, {'id': 372, 'image_count': 134}, {'id': 373, 'image_count': 312}, {'id': 374, 'image_count': 8}, {'id': 375, 'image_count': 271}, {'id': 376, 'image_count': 2}, {'id': 377, 'image_count': 103}, {'id': 378, 'image_count': 1938}, {'id': 379, 'image_count': 574}, {'id': 380, 'image_count': 120}, {'id': 381, 'image_count': 2}, {'id': 382, 'image_count': 2}, {'id': 383, 'image_count': 13}, {'id': 384, 'image_count': 29}, {'id': 385, 'image_count': 1710}, {'id': 386, 'image_count': 66}, {'id': 387, 'image_count': 1008}, {'id': 388, 'image_count': 1}, {'id': 389, 'image_count': 3}, {'id': 390, 'image_count': 1942}, {'id': 391, 'image_count': 19}, {'id': 392, 'image_count': 1488}, {'id': 393, 'image_count': 46}, {'id': 394, 'image_count': 106}, {'id': 395, 'image_count': 115}, {'id': 396, 'image_count': 19}, {'id': 397, 'image_count': 2}, {'id': 398, 'image_count': 1}, {'id': 399, 'image_count': 28}, {'id': 400, 'image_count': 9}, {'id': 401, 'image_count': 192}, {'id': 402, 'image_count': 12}, {'id': 403, 'image_count': 21}, {'id': 404, 'image_count': 247}, {'id': 405, 'image_count': 6}, {'id': 406, 'image_count': 64}, {'id': 407, 'image_count': 7}, {'id': 408, 'image_count': 40}, {'id': 409, 'image_count': 542}, {'id': 410, 'image_count': 2}, {'id': 411, 'image_count': 1898}, {'id': 412, 'image_count': 36}, {'id': 413, 'image_count': 4}, {'id': 414, 'image_count': 1}, {'id': 415, 'image_count': 191}, {'id': 416, 'image_count': 6}, {'id': 417, 'image_count': 41}, {'id': 418, 'image_count': 39}, {'id': 419, 'image_count': 46}, {'id': 420, 'image_count': 1}, {'id': 421, 'image_count': 1451}, {'id': 422, 'image_count': 1878}, {'id': 423, 'image_count': 11}, {'id': 424, 'image_count': 82}, {'id': 425, 'image_count': 18}, {'id': 426, 'image_count': 1}, {'id': 427, 'image_count': 7}, {'id': 428, 'image_count': 3}, {'id': 429, 'image_count': 575}, {'id': 430, 'image_count': 1907}, {'id': 431, 'image_count': 8}, {'id': 432, 'image_count': 4}, {'id': 433, 'image_count': 32}, {'id': 434, 'image_count': 11}, {'id': 435, 'image_count': 4}, {'id': 436, 'image_count': 54}, {'id': 437, 'image_count': 202}, {'id': 438, 'image_count': 32}, {'id': 439, 'image_count': 3}, {'id': 440, 'image_count': 130}, {'id': 441, 'image_count': 119}, {'id': 442, 'image_count': 141}, {'id': 443, 'image_count': 29}, {'id': 444, 'image_count': 525}, {'id': 445, 'image_count': 1323}, {'id': 446, 'image_count': 2}, {'id': 447, 'image_count': 113}, {'id': 448, 'image_count': 16}, {'id': 449, 'image_count': 7}, {'id': 450, 'image_count': 35}, {'id': 451, 'image_count': 1908}, {'id': 452, 'image_count': 353}, {'id': 453, 'image_count': 18}, {'id': 454, 'image_count': 14}, {'id': 455, 'image_count': 77}, {'id': 456, 'image_count': 8}, {'id': 457, 'image_count': 37}, {'id': 458, 'image_count': 1}, {'id': 459, 'image_count': 346}, {'id': 460, 'image_count': 19}, {'id': 461, 'image_count': 1779}, {'id': 462, 'image_count': 23}, {'id': 463, 'image_count': 25}, {'id': 464, 'image_count': 67}, {'id': 465, 'image_count': 19}, {'id': 466, 'image_count': 28}, {'id': 467, 'image_count': 4}, {'id': 468, 'image_count': 27}, {'id': 469, 'image_count': 1861}, {'id': 470, 'image_count': 11}, {'id': 471, 'image_count': 13}, {'id': 472, 'image_count': 13}, {'id': 473, 'image_count': 32}, {'id': 474, 'image_count': 1767}, {'id': 475, 'image_count': 42}, {'id': 476, 'image_count': 17}, {'id': 477, 'image_count': 128}, {'id': 478, 'image_count': 1}, {'id': 479, 'image_count': 9}, {'id': 480, 'image_count': 10}, {'id': 481, 'image_count': 4}, {'id': 482, 'image_count': 9}, {'id': 483, 'image_count': 18}, {'id': 484, 'image_count': 41}, {'id': 485, 'image_count': 28}, {'id': 486, 'image_count': 3}, {'id': 487, 'image_count': 65}, {'id': 488, 'image_count': 9}, {'id': 489, 'image_count': 23}, {'id': 490, 'image_count': 24}, {'id': 491, 'image_count': 1}, {'id': 492, 'image_count': 2}, {'id': 493, 'image_count': 59}, {'id': 494, 'image_count': 48}, {'id': 495, 'image_count': 17}, {'id': 496, 'image_count': 1877}, {'id': 497, 'image_count': 18}, {'id': 498, 'image_count': 1920}, {'id': 499, 'image_count': 50}, {'id': 500, 'image_count': 1890}, {'id': 501, 'image_count': 99}, {'id': 502, 'image_count': 1530}, {'id': 503, 'image_count': 3}, {'id': 504, 'image_count': 11}, {'id': 505, 'image_count': 19}, {'id': 506, 'image_count': 3}, {'id': 507, 'image_count': 63}, {'id': 508, 'image_count': 5}, {'id': 509, 'image_count': 6}, {'id': 510, 'image_count': 233}, {'id': 511, 'image_count': 54}, {'id': 512, 'image_count': 36}, {'id': 513, 'image_count': 10}, {'id': 514, 'image_count': 124}, {'id': 515, 'image_count': 101}, {'id': 516, 'image_count': 3}, {'id': 517, 'image_count': 363}, {'id': 518, 'image_count': 3}, {'id': 519, 'image_count': 30}, {'id': 520, 'image_count': 18}, {'id': 521, 'image_count': 199}, {'id': 522, 'image_count': 97}, {'id': 523, 'image_count': 32}, {'id': 524, 'image_count': 121}, {'id': 525, 'image_count': 16}, {'id': 526, 'image_count': 12}, {'id': 527, 'image_count': 2}, {'id': 528, 'image_count': 214}, {'id': 529, 'image_count': 48}, {'id': 530, 'image_count': 26}, {'id': 531, 'image_count': 13}, {'id': 532, 'image_count': 4}, {'id': 533, 'image_count': 11}, {'id': 534, 'image_count': 123}, {'id': 535, 'image_count': 7}, {'id': 536, 'image_count': 200}, {'id': 537, 'image_count': 91}, {'id': 538, 'image_count': 9}, {'id': 539, 'image_count': 72}, {'id': 540, 'image_count': 1886}, {'id': 541, 'image_count': 4}, {'id': 542, 'image_count': 1}, {'id': 543, 'image_count': 1}, {'id': 544, 'image_count': 1932}, {'id': 545, 'image_count': 4}, {'id': 546, 'image_count': 56}, {'id': 547, 'image_count': 854}, {'id': 548, 'image_count': 755}, {'id': 549, 'image_count': 1843}, {'id': 550, 'image_count': 96}, {'id': 551, 'image_count': 7}, {'id': 552, 'image_count': 74}, {'id': 553, 'image_count': 66}, {'id': 554, 'image_count': 57}, {'id': 555, 'image_count': 44}, {'id': 556, 'image_count': 1905}, {'id': 557, 'image_count': 4}, {'id': 558, 'image_count': 90}, {'id': 559, 'image_count': 1635}, {'id': 560, 'image_count': 8}, {'id': 561, 'image_count': 5}, {'id': 562, 'image_count': 50}, {'id': 563, 'image_count': 545}, {'id': 564, 'image_count': 20}, {'id': 565, 'image_count': 193}, {'id': 566, 'image_count': 285}, {'id': 567, 'image_count': 3}, {'id': 568, 'image_count': 1}, {'id': 569, 'image_count': 1904}, {'id': 570, 'image_count': 294}, {'id': 571, 'image_count': 3}, {'id': 572, 'image_count': 5}, {'id': 573, 'image_count': 24}, {'id': 574, 'image_count': 2}, {'id': 575, 'image_count': 2}, {'id': 576, 'image_count': 16}, {'id': 577, 'image_count': 8}, {'id': 578, 'image_count': 154}, {'id': 579, 'image_count': 66}, {'id': 580, 'image_count': 1}, {'id': 581, 'image_count': 24}, {'id': 582, 'image_count': 1}, {'id': 583, 'image_count': 4}, {'id': 584, 'image_count': 75}, {'id': 585, 'image_count': 6}, {'id': 586, 'image_count': 126}, {'id': 587, 'image_count': 24}, {'id': 588, 'image_count': 22}, {'id': 589, 'image_count': 1872}, {'id': 590, 'image_count': 16}, {'id': 591, 'image_count': 423}, {'id': 592, 'image_count': 1927}, {'id': 593, 'image_count': 38}, {'id': 594, 'image_count': 3}, {'id': 595, 'image_count': 1945}, {'id': 596, 'image_count': 35}, {'id': 597, 'image_count': 1}, {'id': 598, 'image_count': 13}, {'id': 599, 'image_count': 9}, {'id': 600, 'image_count': 14}, {'id': 601, 'image_count': 37}, {'id': 602, 'image_count': 3}, {'id': 603, 'image_count': 4}, {'id': 604, 'image_count': 100}, {'id': 605, 'image_count': 195}, {'id': 606, 'image_count': 1}, {'id': 607, 'image_count': 12}, {'id': 608, 'image_count': 24}, {'id': 609, 'image_count': 489}, {'id': 610, 'image_count': 10}, {'id': 611, 'image_count': 1689}, {'id': 612, 'image_count': 42}, {'id': 613, 'image_count': 81}, {'id': 614, 'image_count': 894}, {'id': 615, 'image_count': 1868}, {'id': 616, 'image_count': 7}, {'id': 617, 'image_count': 1567}, {'id': 618, 'image_count': 10}, {'id': 619, 'image_count': 8}, {'id': 620, 'image_count': 7}, {'id': 621, 'image_count': 629}, {'id': 622, 'image_count': 89}, {'id': 623, 'image_count': 15}, {'id': 624, 'image_count': 134}, {'id': 625, 'image_count': 4}, {'id': 626, 'image_count': 1802}, {'id': 627, 'image_count': 595}, {'id': 628, 'image_count': 1210}, {'id': 629, 'image_count': 48}, {'id': 630, 'image_count': 418}, {'id': 631, 'image_count': 1846}, {'id': 632, 'image_count': 5}, {'id': 633, 'image_count': 221}, {'id': 634, 'image_count': 10}, {'id': 635, 'image_count': 7}, {'id': 636, 'image_count': 76}, {'id': 637, 'image_count': 22}, {'id': 638, 'image_count': 10}, {'id': 639, 'image_count': 341}, {'id': 640, 'image_count': 1}, {'id': 641, 'image_count': 705}, {'id': 642, 'image_count': 1900}, {'id': 643, 'image_count': 188}, {'id': 644, 'image_count': 227}, {'id': 645, 'image_count': 861}, {'id': 646, 'image_count': 6}, {'id': 647, 'image_count': 115}, {'id': 648, 'image_count': 5}, {'id': 649, 'image_count': 43}, {'id': 650, 'image_count': 14}, {'id': 651, 'image_count': 6}, {'id': 652, 'image_count': 15}, {'id': 653, 'image_count': 1167}, {'id': 654, 'image_count': 15}, {'id': 655, 'image_count': 994}, {'id': 656, 'image_count': 28}, {'id': 657, 'image_count': 2}, {'id': 658, 'image_count': 338}, {'id': 659, 'image_count': 334}, {'id': 660, 'image_count': 15}, {'id': 661, 'image_count': 102}, {'id': 662, 'image_count': 1}, {'id': 663, 'image_count': 8}, {'id': 664, 'image_count': 1}, {'id': 665, 'image_count': 1}, {'id': 666, 'image_count': 28}, {'id': 667, 'image_count': 91}, {'id': 668, 'image_count': 260}, {'id': 669, 'image_count': 131}, {'id': 670, 'image_count': 128}, {'id': 671, 'image_count': 3}, {'id': 672, 'image_count': 10}, {'id': 673, 'image_count': 39}, {'id': 674, 'image_count': 2}, {'id': 675, 'image_count': 925}, {'id': 676, 'image_count': 354}, {'id': 677, 'image_count': 31}, {'id': 678, 'image_count': 10}, {'id': 679, 'image_count': 215}, {'id': 680, 'image_count': 71}, {'id': 681, 'image_count': 43}, {'id': 682, 'image_count': 28}, {'id': 683, 'image_count': 34}, {'id': 684, 'image_count': 16}, {'id': 685, 'image_count': 273}, {'id': 686, 'image_count': 2}, {'id': 687, 'image_count': 999}, {'id': 688, 'image_count': 4}, {'id': 689, 'image_count': 107}, {'id': 690, 'image_count': 2}, {'id': 691, 'image_count': 1}, {'id': 692, 'image_count': 454}, {'id': 693, 'image_count': 9}, {'id': 694, 'image_count': 1901}, {'id': 695, 'image_count': 61}, {'id': 696, 'image_count': 91}, {'id': 697, 'image_count': 46}, {'id': 698, 'image_count': 1402}, {'id': 699, 'image_count': 74}, {'id': 700, 'image_count': 421}, {'id': 701, 'image_count': 226}, {'id': 702, 'image_count': 10}, {'id': 703, 'image_count': 1720}, {'id': 704, 'image_count': 261}, {'id': 705, 'image_count': 1337}, {'id': 706, 'image_count': 293}, {'id': 707, 'image_count': 62}, {'id': 708, 'image_count': 814}, {'id': 709, 'image_count': 407}, {'id': 710, 'image_count': 6}, {'id': 711, 'image_count': 16}, {'id': 712, 'image_count': 7}, {'id': 713, 'image_count': 1791}, {'id': 714, 'image_count': 2}, {'id': 715, 'image_count': 1915}, {'id': 716, 'image_count': 1940}, {'id': 717, 'image_count': 13}, {'id': 718, 'image_count': 16}, {'id': 719, 'image_count': 448}, {'id': 720, 'image_count': 12}, {'id': 721, 'image_count': 18}, {'id': 722, 'image_count': 4}, {'id': 723, 'image_count': 71}, {'id': 724, 'image_count': 189}, {'id': 725, 'image_count': 74}, {'id': 726, 'image_count': 103}, {'id': 727, 'image_count': 3}, {'id': 728, 'image_count': 110}, {'id': 729, 'image_count': 5}, {'id': 730, 'image_count': 9}, {'id': 731, 'image_count': 15}, {'id': 732, 'image_count': 25}, {'id': 733, 'image_count': 7}, {'id': 734, 'image_count': 647}, {'id': 735, 'image_count': 824}, {'id': 736, 'image_count': 100}, {'id': 737, 'image_count': 47}, {'id': 738, 'image_count': 121}, {'id': 739, 'image_count': 731}, {'id': 740, 'image_count': 73}, {'id': 741, 'image_count': 49}, {'id': 742, 'image_count': 23}, {'id': 743, 'image_count': 4}, {'id': 744, 'image_count': 62}, {'id': 745, 'image_count': 118}, {'id': 746, 'image_count': 99}, {'id': 747, 'image_count': 40}, {'id': 748, 'image_count': 1036}, {'id': 749, 'image_count': 105}, {'id': 750, 'image_count': 21}, {'id': 751, 'image_count': 229}, {'id': 752, 'image_count': 7}, {'id': 753, 'image_count': 72}, {'id': 754, 'image_count': 9}, {'id': 755, 'image_count': 10}, {'id': 756, 'image_count': 328}, {'id': 757, 'image_count': 468}, {'id': 758, 'image_count': 1}, {'id': 759, 'image_count': 2}, {'id': 760, 'image_count': 24}, {'id': 761, 'image_count': 11}, {'id': 762, 'image_count': 72}, {'id': 763, 'image_count': 17}, {'id': 764, 'image_count': 10}, {'id': 765, 'image_count': 17}, {'id': 766, 'image_count': 489}, {'id': 767, 'image_count': 47}, {'id': 768, 'image_count': 93}, {'id': 769, 'image_count': 1}, {'id': 770, 'image_count': 12}, {'id': 771, 'image_count': 228}, {'id': 772, 'image_count': 5}, {'id': 773, 'image_count': 76}, {'id': 774, 'image_count': 71}, {'id': 775, 'image_count': 30}, {'id': 776, 'image_count': 109}, {'id': 777, 'image_count': 14}, {'id': 778, 'image_count': 1}, {'id': 779, 'image_count': 8}, {'id': 780, 'image_count': 26}, {'id': 781, 'image_count': 339}, {'id': 782, 'image_count': 153}, {'id': 783, 'image_count': 2}, {'id': 784, 'image_count': 3}, {'id': 785, 'image_count': 8}, {'id': 786, 'image_count': 47}, {'id': 787, 'image_count': 8}, {'id': 788, 'image_count': 6}, {'id': 789, 'image_count': 116}, {'id': 790, 'image_count': 69}, {'id': 791, 'image_count': 13}, {'id': 792, 'image_count': 6}, {'id': 793, 'image_count': 1928}, {'id': 794, 'image_count': 79}, {'id': 795, 'image_count': 14}, {'id': 796, 'image_count': 7}, {'id': 797, 'image_count': 20}, {'id': 798, 'image_count': 114}, {'id': 799, 'image_count': 221}, {'id': 800, 'image_count': 502}, {'id': 801, 'image_count': 62}, {'id': 802, 'image_count': 87}, {'id': 803, 'image_count': 4}, {'id': 804, 'image_count': 1912}, {'id': 805, 'image_count': 7}, {'id': 806, 'image_count': 186}, {'id': 807, 'image_count': 18}, {'id': 808, 'image_count': 4}, {'id': 809, 'image_count': 3}, {'id': 810, 'image_count': 7}, {'id': 811, 'image_count': 1413}, {'id': 812, 'image_count': 7}, {'id': 813, 'image_count': 12}, {'id': 814, 'image_count': 248}, {'id': 815, 'image_count': 4}, {'id': 816, 'image_count': 1881}, {'id': 817, 'image_count': 529}, {'id': 818, 'image_count': 1932}, {'id': 819, 'image_count': 50}, {'id': 820, 'image_count': 3}, {'id': 821, 'image_count': 28}, {'id': 822, 'image_count': 10}, {'id': 823, 'image_count': 5}, {'id': 824, 'image_count': 5}, {'id': 825, 'image_count': 18}, {'id': 826, 'image_count': 14}, {'id': 827, 'image_count': 1890}, {'id': 828, 'image_count': 660}, {'id': 829, 'image_count': 8}, {'id': 830, 'image_count': 25}, {'id': 831, 'image_count': 10}, {'id': 832, 'image_count': 218}, {'id': 833, 'image_count': 36}, {'id': 834, 'image_count': 16}, {'id': 835, 'image_count': 808}, {'id': 836, 'image_count': 479}, {'id': 837, 'image_count': 1404}, {'id': 838, 'image_count': 307}, {'id': 839, 'image_count': 57}, {'id': 840, 'image_count': 28}, {'id': 841, 'image_count': 80}, {'id': 842, 'image_count': 11}, {'id': 843, 'image_count': 92}, {'id': 844, 'image_count': 20}, {'id': 845, 'image_count': 194}, {'id': 846, 'image_count': 23}, {'id': 847, 'image_count': 52}, {'id': 848, 'image_count': 673}, {'id': 849, 'image_count': 2}, {'id': 850, 'image_count': 2}, {'id': 851, 'image_count': 1}, {'id': 852, 'image_count': 2}, {'id': 853, 'image_count': 8}, {'id': 854, 'image_count': 80}, {'id': 855, 'image_count': 3}, {'id': 856, 'image_count': 3}, {'id': 857, 'image_count': 15}, {'id': 858, 'image_count': 2}, {'id': 859, 'image_count': 10}, {'id': 860, 'image_count': 386}, {'id': 861, 'image_count': 65}, {'id': 862, 'image_count': 3}, {'id': 863, 'image_count': 35}, {'id': 864, 'image_count': 5}, {'id': 865, 'image_count': 180}, {'id': 866, 'image_count': 99}, {'id': 867, 'image_count': 49}, {'id': 868, 'image_count': 28}, {'id': 869, 'image_count': 1}, {'id': 870, 'image_count': 52}, {'id': 871, 'image_count': 36}, {'id': 872, 'image_count': 70}, {'id': 873, 'image_count': 6}, {'id': 874, 'image_count': 29}, {'id': 875, 'image_count': 24}, {'id': 876, 'image_count': 1115}, {'id': 877, 'image_count': 61}, {'id': 878, 'image_count': 18}, {'id': 879, 'image_count': 18}, {'id': 880, 'image_count': 665}, {'id': 881, 'image_count': 1096}, {'id': 882, 'image_count': 29}, {'id': 883, 'image_count': 8}, {'id': 884, 'image_count': 14}, {'id': 885, 'image_count': 1622}, {'id': 886, 'image_count': 2}, {'id': 887, 'image_count': 3}, {'id': 888, 'image_count': 32}, {'id': 889, 'image_count': 55}, {'id': 890, 'image_count': 1}, {'id': 891, 'image_count': 10}, {'id': 892, 'image_count': 10}, {'id': 893, 'image_count': 47}, {'id': 894, 'image_count': 3}, {'id': 895, 'image_count': 29}, {'id': 896, 'image_count': 342}, {'id': 897, 'image_count': 25}, {'id': 898, 'image_count': 1469}, {'id': 899, 'image_count': 521}, {'id': 900, 'image_count': 347}, {'id': 901, 'image_count': 35}, {'id': 902, 'image_count': 7}, {'id': 903, 'image_count': 207}, {'id': 904, 'image_count': 108}, {'id': 905, 'image_count': 2}, {'id': 906, 'image_count': 34}, {'id': 907, 'image_count': 12}, {'id': 908, 'image_count': 10}, {'id': 909, 'image_count': 13}, {'id': 910, 'image_count': 361}, {'id': 911, 'image_count': 1023}, {'id': 912, 'image_count': 782}, {'id': 913, 'image_count': 2}, {'id': 914, 'image_count': 5}, {'id': 915, 'image_count': 247}, {'id': 916, 'image_count': 221}, {'id': 917, 'image_count': 4}, {'id': 918, 'image_count': 8}, {'id': 919, 'image_count': 158}, {'id': 920, 'image_count': 3}, {'id': 921, 'image_count': 752}, {'id': 922, 'image_count': 64}, {'id': 923, 'image_count': 707}, {'id': 924, 'image_count': 143}, {'id': 925, 'image_count': 1}, {'id': 926, 'image_count': 49}, {'id': 927, 'image_count': 126}, {'id': 928, 'image_count': 76}, {'id': 929, 'image_count': 11}, {'id': 930, 'image_count': 11}, {'id': 931, 'image_count': 4}, {'id': 932, 'image_count': 39}, {'id': 933, 'image_count': 11}, {'id': 934, 'image_count': 13}, {'id': 935, 'image_count': 91}, {'id': 936, 'image_count': 14}, {'id': 937, 'image_count': 5}, {'id': 938, 'image_count': 3}, {'id': 939, 'image_count': 10}, {'id': 940, 'image_count': 18}, {'id': 941, 'image_count': 9}, {'id': 942, 'image_count': 6}, {'id': 943, 'image_count': 951}, {'id': 944, 'image_count': 2}, {'id': 945, 'image_count': 1}, {'id': 946, 'image_count': 19}, {'id': 947, 'image_count': 1942}, {'id': 948, 'image_count': 1916}, {'id': 949, 'image_count': 139}, {'id': 950, 'image_count': 43}, {'id': 951, 'image_count': 1969}, {'id': 952, 'image_count': 5}, {'id': 953, 'image_count': 134}, {'id': 954, 'image_count': 74}, {'id': 955, 'image_count': 381}, {'id': 956, 'image_count': 1}, {'id': 957, 'image_count': 381}, {'id': 958, 'image_count': 6}, {'id': 959, 'image_count': 1826}, {'id': 960, 'image_count': 28}, {'id': 961, 'image_count': 1635}, {'id': 962, 'image_count': 1967}, {'id': 963, 'image_count': 16}, {'id': 964, 'image_count': 1926}, {'id': 965, 'image_count': 1789}, {'id': 966, 'image_count': 401}, {'id': 967, 'image_count': 1968}, {'id': 968, 'image_count': 1167}, {'id': 969, 'image_count': 1}, {'id': 970, 'image_count': 56}, {'id': 971, 'image_count': 17}, {'id': 972, 'image_count': 1}, {'id': 973, 'image_count': 58}, {'id': 974, 'image_count': 9}, {'id': 975, 'image_count': 8}, {'id': 976, 'image_count': 1124}, {'id': 977, 'image_count': 31}, {'id': 978, 'image_count': 16}, {'id': 979, 'image_count': 491}, {'id': 980, 'image_count': 432}, {'id': 981, 'image_count': 1945}, {'id': 982, 'image_count': 1899}, {'id': 983, 'image_count': 5}, {'id': 984, 'image_count': 28}, {'id': 985, 'image_count': 7}, {'id': 986, 'image_count': 146}, {'id': 987, 'image_count': 1}, {'id': 988, 'image_count': 25}, {'id': 989, 'image_count': 22}, {'id': 990, 'image_count': 1}, {'id': 991, 'image_count': 10}, {'id': 992, 'image_count': 9}, {'id': 993, 'image_count': 308}, {'id': 994, 'image_count': 4}, {'id': 995, 'image_count': 1969}, {'id': 996, 'image_count': 45}, {'id': 997, 'image_count': 12}, {'id': 998, 'image_count': 1}, {'id': 999, 'image_count': 85}, {'id': 1000, 'image_count': 1127}, {'id': 1001, 'image_count': 11}, {'id': 1002, 'image_count': 60}, {'id': 1003, 'image_count': 1}, {'id': 1004, 'image_count': 16}, {'id': 1005, 'image_count': 1}, {'id': 1006, 'image_count': 65}, {'id': 1007, 'image_count': 13}, {'id': 1008, 'image_count': 655}, {'id': 1009, 'image_count': 51}, {'id': 1010, 'image_count': 1}, {'id': 1011, 'image_count': 673}, {'id': 1012, 'image_count': 5}, {'id': 1013, 'image_count': 36}, {'id': 1014, 'image_count': 54}, {'id': 1015, 'image_count': 5}, {'id': 1016, 'image_count': 8}, {'id': 1017, 'image_count': 305}, {'id': 1018, 'image_count': 297}, {'id': 1019, 'image_count': 1053}, {'id': 1020, 'image_count': 223}, {'id': 1021, 'image_count': 1037}, {'id': 1022, 'image_count': 63}, {'id': 1023, 'image_count': 1881}, {'id': 1024, 'image_count': 507}, {'id': 1025, 'image_count': 333}, {'id': 1026, 'image_count': 1911}, {'id': 1027, 'image_count': 1765}, {'id': 1028, 'image_count': 1}, {'id': 1029, 'image_count': 5}, {'id': 1030, 'image_count': 1}, {'id': 1031, 'image_count': 9}, {'id': 1032, 'image_count': 2}, {'id': 1033, 'image_count': 151}, {'id': 1034, 'image_count': 82}, {'id': 1035, 'image_count': 1931}, {'id': 1036, 'image_count': 41}, {'id': 1037, 'image_count': 1895}, {'id': 1038, 'image_count': 24}, {'id': 1039, 'image_count': 22}, {'id': 1040, 'image_count': 35}, {'id': 1041, 'image_count': 69}, {'id': 1042, 'image_count': 962}, {'id': 1043, 'image_count': 588}, {'id': 1044, 'image_count': 21}, {'id': 1045, 'image_count': 825}, {'id': 1046, 'image_count': 52}, {'id': 1047, 'image_count': 5}, {'id': 1048, 'image_count': 5}, {'id': 1049, 'image_count': 5}, {'id': 1050, 'image_count': 1860}, {'id': 1051, 'image_count': 56}, {'id': 1052, 'image_count': 1582}, {'id': 1053, 'image_count': 7}, {'id': 1054, 'image_count': 2}, {'id': 1055, 'image_count': 1562}, {'id': 1056, 'image_count': 1885}, {'id': 1057, 'image_count': 1}, {'id': 1058, 'image_count': 5}, {'id': 1059, 'image_count': 137}, {'id': 1060, 'image_count': 1094}, {'id': 1061, 'image_count': 134}, {'id': 1062, 'image_count': 29}, {'id': 1063, 'image_count': 22}, {'id': 1064, 'image_count': 522}, {'id': 1065, 'image_count': 50}, {'id': 1066, 'image_count': 68}, {'id': 1067, 'image_count': 16}, {'id': 1068, 'image_count': 40}, {'id': 1069, 'image_count': 35}, {'id': 1070, 'image_count': 135}, {'id': 1071, 'image_count': 1413}, {'id': 1072, 'image_count': 772}, {'id': 1073, 'image_count': 50}, {'id': 1074, 'image_count': 1015}, {'id': 1075, 'image_count': 1}, {'id': 1076, 'image_count': 65}, {'id': 1077, 'image_count': 1900}, {'id': 1078, 'image_count': 1302}, {'id': 1079, 'image_count': 1977}, {'id': 1080, 'image_count': 2}, {'id': 1081, 'image_count': 29}, {'id': 1082, 'image_count': 36}, {'id': 1083, 'image_count': 138}, {'id': 1084, 'image_count': 4}, {'id': 1085, 'image_count': 67}, {'id': 1086, 'image_count': 26}, {'id': 1087, 'image_count': 25}, {'id': 1088, 'image_count': 33}, {'id': 1089, 'image_count': 37}, {'id': 1090, 'image_count': 50}, {'id': 1091, 'image_count': 270}, {'id': 1092, 'image_count': 12}, {'id': 1093, 'image_count': 316}, {'id': 1094, 'image_count': 41}, {'id': 1095, 'image_count': 224}, {'id': 1096, 'image_count': 105}, {'id': 1097, 'image_count': 1925}, {'id': 1098, 'image_count': 1021}, {'id': 1099, 'image_count': 1213}, {'id': 1100, 'image_count': 172}, {'id': 1101, 'image_count': 28}, {'id': 1102, 'image_count': 745}, {'id': 1103, 'image_count': 187}, {'id': 1104, 'image_count': 147}, {'id': 1105, 'image_count': 136}, {'id': 1106, 'image_count': 34}, {'id': 1107, 'image_count': 41}, {'id': 1108, 'image_count': 636}, {'id': 1109, 'image_count': 570}, {'id': 1110, 'image_count': 1149}, {'id': 1111, 'image_count': 61}, {'id': 1112, 'image_count': 1890}, {'id': 1113, 'image_count': 18}, {'id': 1114, 'image_count': 143}, {'id': 1115, 'image_count': 1517}, {'id': 1116, 'image_count': 7}, {'id': 1117, 'image_count': 943}, {'id': 1118, 'image_count': 6}, {'id': 1119, 'image_count': 1}, {'id': 1120, 'image_count': 11}, {'id': 1121, 'image_count': 101}, {'id': 1122, 'image_count': 1909}, {'id': 1123, 'image_count': 800}, {'id': 1124, 'image_count': 1}, {'id': 1125, 'image_count': 44}, {'id': 1126, 'image_count': 3}, {'id': 1127, 'image_count': 44}, {'id': 1128, 'image_count': 31}, {'id': 1129, 'image_count': 7}, {'id': 1130, 'image_count': 20}, {'id': 1131, 'image_count': 11}, {'id': 1132, 'image_count': 13}, {'id': 1133, 'image_count': 1924}, {'id': 1134, 'image_count': 113}, {'id': 1135, 'image_count': 2}, {'id': 1136, 'image_count': 139}, {'id': 1137, 'image_count': 12}, {'id': 1138, 'image_count': 37}, {'id': 1139, 'image_count': 1866}, {'id': 1140, 'image_count': 47}, {'id': 1141, 'image_count': 1468}, {'id': 1142, 'image_count': 729}, {'id': 1143, 'image_count': 24}, {'id': 1144, 'image_count': 1}, {'id': 1145, 'image_count': 10}, {'id': 1146, 'image_count': 3}, {'id': 1147, 'image_count': 14}, {'id': 1148, 'image_count': 4}, {'id': 1149, 'image_count': 29}, {'id': 1150, 'image_count': 4}, {'id': 1151, 'image_count': 70}, {'id': 1152, 'image_count': 46}, {'id': 1153, 'image_count': 14}, {'id': 1154, 'image_count': 48}, {'id': 1155, 'image_count': 1855}, {'id': 1156, 'image_count': 113}, {'id': 1157, 'image_count': 1}, {'id': 1158, 'image_count': 1}, {'id': 1159, 'image_count': 10}, {'id': 1160, 'image_count': 54}, {'id': 1161, 'image_count': 1923}, {'id': 1162, 'image_count': 630}, {'id': 1163, 'image_count': 31}, {'id': 1164, 'image_count': 69}, {'id': 1165, 'image_count': 7}, {'id': 1166, 'image_count': 11}, {'id': 1167, 'image_count': 1}, {'id': 1168, 'image_count': 30}, {'id': 1169, 'image_count': 50}, {'id': 1170, 'image_count': 45}, {'id': 1171, 'image_count': 28}, {'id': 1172, 'image_count': 114}, {'id': 1173, 'image_count': 193}, {'id': 1174, 'image_count': 21}, {'id': 1175, 'image_count': 91}, {'id': 1176, 'image_count': 31}, {'id': 1177, 'image_count': 1469}, {'id': 1178, 'image_count': 1924}, {'id': 1179, 'image_count': 87}, {'id': 1180, 'image_count': 77}, {'id': 1181, 'image_count': 11}, {'id': 1182, 'image_count': 47}, {'id': 1183, 'image_count': 21}, {'id': 1184, 'image_count': 47}, {'id': 1185, 'image_count': 70}, {'id': 1186, 'image_count': 1838}, {'id': 1187, 'image_count': 19}, {'id': 1188, 'image_count': 531}, {'id': 1189, 'image_count': 11}, {'id': 1190, 'image_count': 941}, {'id': 1191, 'image_count': 113}, {'id': 1192, 'image_count': 26}, {'id': 1193, 'image_count': 5}, {'id': 1194, 'image_count': 56}, {'id': 1195, 'image_count': 73}, {'id': 1196, 'image_count': 32}, {'id': 1197, 'image_count': 128}, {'id': 1198, 'image_count': 623}, {'id': 1199, 'image_count': 12}, {'id': 1200, 'image_count': 52}, {'id': 1201, 'image_count': 11}, {'id': 1202, 'image_count': 1674}, {'id': 1203, 'image_count': 81}] # noqa diff --git a/perception_models/apps/detection/DETA_pe/models/utils_softnms.py b/perception_models/apps/detection/DETA_pe/models/utils_softnms.py new file mode 100644 index 0000000000000000000000000000000000000000..d390eadc5003e9ffbea17c1344526234da981d6e --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/models/utils_softnms.py @@ -0,0 +1,203 @@ +import torch + +try: + from detectron2.structures import ( + Boxes, + pairwise_iou, + pairwise_iou_rotated, + RotatedBoxes, + ) +except ImportError: + pass + + +def batched_soft_nms( + boxes, + scores, + idxs, + threshold=0.7, + method="linear", + gaussian_sigma=0.5, + prune_threshold=0.001, + quad_scale=1.0, +): + """ + Performs soft non-maximum suppression in a batched fashion. + Each index value correspond to a category, and NMS + will not be applied between elements of different categories. + Args: + boxes (Tensor[N, 4]): + boxes where NMS will be performed. They + are expected to be in (x1, y1, x2, y2) format + scores (Tensor[N]): + scores for each one of the boxes + idxs (Tensor[N]): + indices of the categories for each one of the boxes. + method (str): + one of ['gaussian', 'linear', 'hard'] + see paper for details. users encouraged not to use "hard", as this is the + same nms available elsewhere in detectron2 + gaussian_sigma (float): + parameter for Gaussian penalty function + linear_threshold (float): + iou threshold for applying linear decay. Nt from the paper + re-used as threshold for standard "hard" nms + prune_threshold (float): + boxes with scores below this threshold are pruned at each iteration. + Dramatically reduces computation time. Authors use values in [10e-4, 10e-2] + Returns: + tuple(Tensor, Tensor): + [0]: int64 tensor with the indices of the elements that have been kept + by Soft NMS, sorted in decreasing order of scores + [1]: float tensor with the re-scored scores of the elements that were kept + """ + if boxes.numel() == 0: + return ( + torch.empty((0,), dtype=torch.int64, device=boxes.device), + torch.empty((0,), dtype=torch.float32, device=scores.device), + ) + # strategy: in order to perform NMS independently per class. + # we add an offset to all the boxes. The offset is dependent + # only on the class idx, and is large enough so that boxes + # from different classes do not overlap + max_coordinate = boxes.max() + offsets = idxs.to(boxes) * (max_coordinate + 1) + boxes_for_nms = boxes + offsets[:, None] + return soft_nms( + boxes_for_nms, + scores, + method, + gaussian_sigma, + threshold, + prune_threshold, + quad_scale, + ) + + +def soft_nms( + boxes, scores, method, gaussian_sigma, linear_threshold, prune_threshold, quad_scale +): + """ + Performs soft non-maximum suppression algorithm on axis aligned boxes + Args: + boxes (Tensor[N, 5]): + boxes where NMS will be performed. They + are expected to be in (x_ctr, y_ctr, width, height, angle_degrees) format + scores (Tensor[N]): + scores for each one of the boxes + method (str): + one of ['gaussian', 'linear', 'hard'] + see paper for details. users encouraged not to use "hard", as this is the + same nms available elsewhere in detectron2 + gaussian_sigma (float): + parameter for Gaussian penalty function + linear_threshold (float): + iou threshold for applying linear decay. Nt from the paper + re-used as threshold for standard "hard" nms + prune_threshold (float): + boxes with scores below this threshold are pruned at each iteration. + Dramatically reduces computation time. Authors use values in [10e-4, 10e-2] + Returns: + tuple(Tensor, Tensor): + [0]: int64 tensor with the indices of the elements that have been kept + by Soft NMS, sorted in decreasing order of scores + [1]: float tensor with the re-scored scores of the elements that were kept + """ + return _soft_nms( + Boxes, + pairwise_iou, + boxes, + scores, + method, + gaussian_sigma, + linear_threshold, + prune_threshold, + quad_scale, + ) + + +def _soft_nms( + box_class, + pairwise_iou_func, + boxes, + scores, + method, + gaussian_sigma, + linear_threshold, + prune_threshold, + quad_scale, +): + """ + Soft non-max suppression algorithm. + Implementation of [Soft-NMS -- Improving Object Detection With One Line of Codec] + (https://arxiv.org/abs/1704.04503) + Args: + box_class (cls): one of Box, RotatedBoxes + pairwise_iou_func (func): one of pairwise_iou, pairwise_iou_rotated + boxes (Tensor[N, ?]): + boxes where NMS will be performed + if Boxes, in (x1, y1, x2, y2) format + if RotatedBoxes, in (x_ctr, y_ctr, width, height, angle_degrees) format + scores (Tensor[N]): + scores for each one of the boxes + method (str): + one of ['gaussian', 'linear', 'hard'] + see paper for details. users encouraged not to use "hard", as this is the + same nms available elsewhere in detectron2 + gaussian_sigma (float): + parameter for Gaussian penalty function + linear_threshold (float): + iou threshold for applying linear decay. Nt from the paper + re-used as threshold for standard "hard" nms + prune_threshold (float): + boxes with scores below this threshold are pruned at each iteration. + Dramatically reduces computation time. Authors use values in [10e-4, 10e-2] + Returns: + tuple(Tensor, Tensor): + [0]: int64 tensor with the indices of the elements that have been kept + by Soft NMS, sorted in decreasing order of scores + [1]: float tensor with the re-scored scores of the elements that were kept + """ + boxes = boxes.clone() + scores = scores.clone() + idxs = torch.arange(scores.size()[0]) + + idxs_out = [] + scores_out = [] + + while scores.numel() > 0: + top_idx = torch.argmax(scores) + idxs_out.append(idxs[top_idx].item()) + scores_out.append(scores[top_idx].item()) + + top_box = boxes[top_idx] + ious = pairwise_iou_func(box_class(top_box.unsqueeze(0)), box_class(boxes))[0] + + if method == "quad": + decay = torch.ones_like(ious) + decay_mask = ious > linear_threshold + decay[decay_mask] = (1 - ious[decay_mask]) ** quad_scale + elif method == "linear": + decay = torch.ones_like(ious) + decay_mask = ious > linear_threshold + decay[decay_mask] = 1 - ious[decay_mask] + elif method == "gaussian": + decay = torch.exp(-torch.pow(ious, 2) / gaussian_sigma) + elif method == "hard": # standard NMS + decay = (ious < linear_threshold).float() + else: + raise NotImplementedError( + "{} soft nms method not implemented.".format(method) + ) + + scores *= decay + keep = scores > prune_threshold + keep[top_idx] = False + + boxes = boxes[keep] + scores = scores[keep] + idxs = idxs[keep.to(idxs.device)] + + return torch.tensor(idxs_out).to(boxes.device), torch.tensor(scores_out).to( + scores.device + ) diff --git a/perception_models/apps/detection/DETA_pe/scripts/eval.sh b/perception_models/apps/detection/DETA_pe/scripts/eval.sh new file mode 100644 index 0000000000000000000000000000000000000000..f7b9f6da70d5aa99ac2af843a74407d0aa14006d --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/scripts/eval.sh @@ -0,0 +1,25 @@ + +EXP_DIR="/checkpoint/vision_encoder/d2_output/coco_sota/eval" + + +python -m torch.distributed.launch --nproc_per_node=8 \ +--master_port=12345 --use_env main.py \ +--output_dir ${EXP_DIR} \ +--with_box_refine --two_stage \ +--num_feature_levels 5 --num_queries 900 \ +--dim_feedforward 2048 --dropout 0.0 --cls_loss_coef 1.0 \ +--assign_first_stage --assign_second_stage \ +--epochs 24 --lr_drop 20 \ +--lr 5e-5 --lr_backbone 5e-5 --batch_size 1 \ +--backbone pev1 \ +--backbone_size Gwin384 \ +--backbone_init_values 0.1 \ +--backbone_tile_posemb True \ +--backbone_lrd 0.9 --backbone_layers 50 \ +--num_workers 4 \ +--coco_path /checkpoint/vision_encoder/public_data/coco \ +--lsj --lsj_img_size 1728 \ +--backbone_use_act_checkpoint --backbone_act_checkpoint_ratio 1.0 \ +--eval \ +--resume /checkpoint/vision_encoder/d2_output/coco_sota/finetune_spatial_Gwin384_cocoep12_1728pix_8node/checkpoint.pth \ +"$@" diff --git a/perception_models/apps/detection/DETA_pe/scripts/eval_1824pix.sh b/perception_models/apps/detection/DETA_pe/scripts/eval_1824pix.sh new file mode 100644 index 0000000000000000000000000000000000000000..a38c5347114518432f67a7e1906f9daf933843ee --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/scripts/eval_1824pix.sh @@ -0,0 +1,25 @@ + +EXP_DIR="/checkpoint/vision_encoder/d2_output/coco_sota/eval" + + +python -m torch.distributed.launch --nproc_per_node=8 \ +--master_port=12345 --use_env main.py \ +--output_dir ${EXP_DIR} \ +--with_box_refine --two_stage \ +--num_feature_levels 5 --num_queries 900 \ +--dim_feedforward 2048 --dropout 0.0 --cls_loss_coef 1.0 \ +--assign_first_stage --assign_second_stage \ +--epochs 24 --lr_drop 20 \ +--lr 5e-5 --lr_backbone 5e-5 --batch_size 1 \ +--backbone pev1 \ +--backbone_size Gwin384 \ +--backbone_init_values 0.1 \ +--backbone_tile_posemb True \ +--backbone_lrd 0.9 --backbone_layers 50 \ +--num_workers 4 \ +--coco_path /checkpoint/vision_encoder/public_data/coco \ +--lsj --lsj_img_size 1824 \ +--backbone_use_act_checkpoint --backbone_act_checkpoint_ratio 1.0 \ +--eval \ +--resume /checkpoint/vision_encoder/d2_output/coco_sota/finetune_further_spatial_Gwin384_cocoep3_1824pix_8node/checkpoint.pth \ +"$@" diff --git a/perception_models/apps/detection/DETA_pe/scripts/eval_tta_slurm.sh b/perception_models/apps/detection/DETA_pe/scripts/eval_tta_slurm.sh new file mode 100644 index 0000000000000000000000000000000000000000..e424f0f918702257e93f497c416e453ffd4ac658 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/scripts/eval_tta_slurm.sh @@ -0,0 +1,60 @@ +#!/bin/bash + +#SBATCH --qos=vision_encoder_high +#SBATCH --account=vision_encoder +#SBATCH --job-name=det +#SBATCH --nodes=8 +#SBATCH --ntasks=8 +#SBATCH --gres=gpu:8 +#SBATCH --cpus-per-task=32 +#SBATCH --mem=0 +#SBATCH --output=/checkpoint/vision_encoder/d2_output/slurm_logs/coco_sota/eval_tta_slurm/%j.out +#SBATCH --error=/checkpoint/vision_encoder/d2_output/slurm_logs/coco_sota/eval_tta_slurm/%j.err +#SBATCH --time=23:00:00 + +module load cuda/12.1 +nodes=($(scontrol show hostnames $SLURM_JOB_NODELIST)) +nodes_array=($nodes) +head_node=${nodes_array[0]} +head_node_ip=$(srun --nodes=1 --ntasks=1 -w "$head_node" hostname --ip-address) + +read -ra my_array <<< $head_node_ip +export LOGLEVEL=INFO + +echo head_node_ip $head_node_ip +echo endpoint "${head_node_ip}:29500" + +EXP_DIR="/checkpoint/vision_encoder/d2_output/coco_sota/eval_tta_slurm" + + +# srun \ +# torchrun \ +srun \ +python -m torch.distributed.run \ +--nnodes 8 \ +--nproc_per_node 8 \ +--rdzv_id $RANDOM \ +--rdzv_endpoint "${my_array[0]}:29500" \ +--rdzv_backend c10d \ +main.py \ +--output_dir ${EXP_DIR} \ +--with_box_refine --two_stage \ +--num_feature_levels 5 --num_queries 2000 \ +--dim_feedforward 2048 --dropout 0.0 --cls_loss_coef 1.0 \ +--assign_first_stage --assign_second_stage \ +--epochs 12 --lr_drop 10 \ +--lr 5e-5 --lr_backbone 5e-5 --batch_size 1 \ +--backbone pev1 \ +--backbone_size Gwin384 \ +--backbone_init_values 0.1 \ +--backbone_tile_posemb True \ +--backbone_lrd 0.9 --backbone_layers 50 \ +--num_workers 4 \ +--coco_path /checkpoint/vision_encoder/public_data/coco \ +--lsj --lsj_img_size 1728 \ +--backbone_use_act_checkpoint --backbone_act_checkpoint_ratio 1.0 \ +--eval \ +--resume /checkpoint/vision_encoder/d2_output/coco_sota/finetune_spatial_Gwin384_cocoep12_1728pix_8node/checkpoint.pth \ +--soft_nms \ +--tta \ +"$@" diff --git a/perception_models/apps/detection/DETA_pe/scripts/eval_tta_slurm_1824pix.sh b/perception_models/apps/detection/DETA_pe/scripts/eval_tta_slurm_1824pix.sh new file mode 100644 index 0000000000000000000000000000000000000000..db96c04e5ed44daba7b8fc42f38dc920510e0e2d --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/scripts/eval_tta_slurm_1824pix.sh @@ -0,0 +1,60 @@ +#!/bin/bash + +#SBATCH --qos=vision_encoder_high +#SBATCH --account=vision_encoder +#SBATCH --job-name=det +#SBATCH --nodes=8 +#SBATCH --ntasks=8 +#SBATCH --gres=gpu:8 +#SBATCH --cpus-per-task=32 +#SBATCH --mem=0 +#SBATCH --output=/checkpoint/vision_encoder/d2_output/slurm_logs/coco_sota/eval_tta_slurm/%j.out +#SBATCH --error=/checkpoint/vision_encoder/d2_output/slurm_logs/coco_sota/eval_tta_slurm/%j.err +#SBATCH --time=23:00:00 + +module load cuda/12.1 +nodes=($(scontrol show hostnames $SLURM_JOB_NODELIST)) +nodes_array=($nodes) +head_node=${nodes_array[0]} +head_node_ip=$(srun --nodes=1 --ntasks=1 -w "$head_node" hostname --ip-address) + +read -ra my_array <<< $head_node_ip +export LOGLEVEL=INFO + +echo head_node_ip $head_node_ip +echo endpoint "${head_node_ip}:29500" + +EXP_DIR="/checkpoint/vision_encoder/d2_output/coco_sota/eval_tta_slurm" + + +# srun \ +# torchrun \ +srun \ +python -m torch.distributed.run \ +--nnodes 8 \ +--nproc_per_node 8 \ +--rdzv_id $RANDOM \ +--rdzv_endpoint "${my_array[0]}:29500" \ +--rdzv_backend c10d \ +main.py \ +--output_dir ${EXP_DIR} \ +--with_box_refine --two_stage \ +--num_feature_levels 5 --num_queries 2000 \ +--dim_feedforward 2048 --dropout 0.0 --cls_loss_coef 1.0 \ +--assign_first_stage --assign_second_stage \ +--epochs 12 --lr_drop 10 \ +--lr 5e-5 --lr_backbone 5e-5 --batch_size 1 \ +--backbone pev1 \ +--backbone_size Gwin384 \ +--backbone_init_values 0.1 \ +--backbone_tile_posemb True \ +--backbone_lrd 0.9 --backbone_layers 50 \ +--num_workers 4 \ +--coco_path /checkpoint/vision_encoder/public_data/coco \ +--lsj --lsj_img_size 1824 \ +--backbone_use_act_checkpoint --backbone_act_checkpoint_ratio 1.0 \ +--eval \ +--resume /checkpoint/vision_encoder/d2_output/coco_sota/finetune_further_spatial_Gwin384_cocoep3_1824pix_8node/checkpoint.pth \ +--soft_nms \ +--tta \ +"$@" diff --git a/perception_models/apps/detection/DETA_pe/scripts/finetune_further_spatial_Gwin384_cocoep3_1824pix_8node.sh b/perception_models/apps/detection/DETA_pe/scripts/finetune_further_spatial_Gwin384_cocoep3_1824pix_8node.sh new file mode 100644 index 0000000000000000000000000000000000000000..8d616bdfc68951f6f8b35ae1c272b6bc2b5f395b --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/scripts/finetune_further_spatial_Gwin384_cocoep3_1824pix_8node.sh @@ -0,0 +1,63 @@ +#!/bin/bash + +#SBATCH --qos=vision_encoder +#SBATCH --account=vision_encoder +#SBATCH --job-name=det +#SBATCH --nodes=8 +#SBATCH --ntasks=8 +#SBATCH --gres=gpu:8 +#SBATCH --cpus-per-task=32 +#SBATCH --mem=0 +#SBATCH --output=/checkpoint/vision_encoder/d2_output/slurm_logs/coco_sota/finetune_further_spatial_Gwin384_cocoep3_1824pix_8node/%j.out +#SBATCH --error=/checkpoint/vision_encoder/d2_output/slurm_logs/coco_sota/finetune_further_spatial_Gwin384_cocoep3_1824pix_8node/%j.err +#SBATCH --time=96:00:00 + +module load cuda/12.1 +nodes=($(scontrol show hostnames $SLURM_JOB_NODELIST)) +nodes_array=($nodes) +head_node=${nodes_array[0]} +head_node_ip=$(srun --nodes=1 --ntasks=1 -w "$head_node" hostname --ip-address) + +read -ra my_array <<< $head_node_ip +export LOGLEVEL=INFO + +echo head_node_ip $head_node_ip +echo endpoint "${head_node_ip}:29500" + +EXP_DIR="/checkpoint/vision_encoder/d2_output/coco_sota/finetune_further_spatial_Gwin384_cocoep3_1824pix_8node" + +export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True + +srun \ +torchrun \ +--nnodes 8 \ +--nproc_per_node 8 \ +--rdzv_id $RANDOM \ +--rdzv_endpoint "${my_array[0]}:29500" \ +--rdzv_backend c10d \ +main.py \ +--output_dir ${EXP_DIR} \ +--with_box_refine --two_stage \ +--num_feature_levels 5 --num_queries 900 \ +--dim_feedforward 2048 --dropout 0.0 --cls_loss_coef 1.0 \ +--assign_first_stage --assign_second_stage \ +--epochs 3 --lr_drop 2 \ +--batch_size 1 \ +--backbone pev1 \ +--backbone_size Gwin384 \ +--backbone_init_values 0.1 \ +--backbone_tile_posemb True \ +--backbone_lrd 0.9 --backbone_layers 50 \ +--coco_path /checkpoint/vision_encoder/public_data/coco \ +--finetune /checkpoint/vision_encoder/d2_output/coco_sota/finetune_spatial_Gwin384_cocoep12_1728pix_8node/checkpoint.pth \ +--lsj --lsj_img_size 1824 \ +--backbone_use_act_checkpoint --backbone_act_checkpoint_ratio 1.0 \ +--eval_per_epochs 1 \ +--save_per_epochs 1 \ +--auto_resume \ +--keep_class_embed \ +--bf16 \ +--backbone_dp 0.0 \ +--sgd \ +--lr 5e-5 --lr_backbone 5e-5 \ +"$@" diff --git a/perception_models/apps/detection/DETA_pe/scripts/finetune_spatial_Gwin384_cocoep12_1728pix_8node.sh b/perception_models/apps/detection/DETA_pe/scripts/finetune_spatial_Gwin384_cocoep12_1728pix_8node.sh new file mode 100644 index 0000000000000000000000000000000000000000..55e78925cf2c0d21a833cc82f54735aed640c945 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/scripts/finetune_spatial_Gwin384_cocoep12_1728pix_8node.sh @@ -0,0 +1,60 @@ +#!/bin/bash + +#SBATCH --qos=vision_encoder +#SBATCH --account=vision_encoder +#SBATCH --job-name=det +#SBATCH --nodes=8 +#SBATCH --ntasks=8 +#SBATCH --gres=gpu:8 +#SBATCH --cpus-per-task=32 +#SBATCH --mem=0 +#SBATCH --output=/checkpoint/vision_encoder/d2_output/slurm_logs/coco_sota/finetune_spatial_Gwin384_cocoep12_1728pix_8node/%j.out +#SBATCH --error=/checkpoint/vision_encoder/d2_output/slurm_logs/coco_sota/finetune_spatial_Gwin384_cocoep12_1728pix_8node/%j.err +#SBATCH --time=96:00:00 + +module load cuda/12.1 +nodes=($(scontrol show hostnames $SLURM_JOB_NODELIST)) +nodes_array=($nodes) +head_node=${nodes_array[0]} +head_node_ip=$(srun --nodes=1 --ntasks=1 -w "$head_node" hostname --ip-address) + +read -ra my_array <<< $head_node_ip +export LOGLEVEL=INFO + +echo head_node_ip $head_node_ip +echo endpoint "${head_node_ip}:29500" + +EXP_DIR="/checkpoint/vision_encoder/d2_output/coco_sota/finetune_spatial_Gwin384_cocoep12_1728pix_8node" + +export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True + +srun \ +torchrun \ +--nnodes 8 \ +--nproc_per_node 8 \ +--rdzv_id $RANDOM \ +--rdzv_endpoint "${my_array[0]}:29500" \ +--rdzv_backend c10d \ +main.py \ +--output_dir ${EXP_DIR} \ +--with_box_refine --two_stage \ +--num_feature_levels 5 --num_queries 900 \ +--dim_feedforward 2048 --dropout 0.0 --cls_loss_coef 1.0 \ +--assign_first_stage --assign_second_stage \ +--epochs 12 --lr_drop 10 \ +--lr 5e-5 --lr_backbone 5e-5 --batch_size 1 \ +--backbone pev1 \ +--backbone_size Gwin384 \ +--backbone_init_values 0.1 \ +--backbone_tile_posemb True \ +--backbone_lrd 0.9 --backbone_layers 50 \ +--coco_path /checkpoint/vision_encoder/public_data/coco \ +--finetune /checkpoint/vision_encoder/d2_output/coco_sota/pretrain_continue_spatial_Gwin384_o365ep6_1536pix_16node/checkpoint.pth \ +--lsj --lsj_img_size 1728 \ +--backbone_use_act_checkpoint --backbone_act_checkpoint_ratio 1.0 \ +--eval_per_epochs 1 \ +--save_per_epochs 1 \ +--auto_resume \ +--bf16 \ +--backbone_dp 0.4 \ +"$@" diff --git a/perception_models/apps/detection/DETA_pe/scripts/pretrain_continue_spatial_Gwin384_o365ep6_1536pix_16node.sh b/perception_models/apps/detection/DETA_pe/scripts/pretrain_continue_spatial_Gwin384_o365ep6_1536pix_16node.sh new file mode 100644 index 0000000000000000000000000000000000000000..c5929b3dd7001ac07dd7b3d73802111c22321d87 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/scripts/pretrain_continue_spatial_Gwin384_o365ep6_1536pix_16node.sh @@ -0,0 +1,58 @@ +#!/bin/bash + +#SBATCH --qos=vision_encoder +#SBATCH --account=vision_encoder +#SBATCH --job-name=det +#SBATCH --nodes=16 +#SBATCH --ntasks=16 +#SBATCH --gres=gpu:8 +#SBATCH --cpus-per-task=32 +#SBATCH --mem=0 +#SBATCH --output=/checkpoint/vision_encoder/d2_output/slurm_logs/coco_sota/pretrain_continue_spatial_Gwin384_o365ep6_1536pix_16node/%j.out +#SBATCH --error=/checkpoint/vision_encoder/d2_output/slurm_logs/coco_sota/pretrain_continue_spatial_Gwin384_o365ep6_1536pix_16node/%j.err +#SBATCH --time=96:00:00 + +module load cuda/12.1 +nodes=($(scontrol show hostnames $SLURM_JOB_NODELIST)) +nodes_array=($nodes) +head_node=${nodes_array[0]} +head_node_ip=$(srun --nodes=1 --ntasks=1 -w "$head_node" hostname --ip-address) + +read -ra my_array <<< $head_node_ip +export LOGLEVEL=INFO + +echo head_node_ip $head_node_ip +echo endpoint "${head_node_ip}:29500" + +EXP_DIR="/checkpoint/vision_encoder/d2_output/coco_sota/pretrain_continue_spatial_Gwin384_o365ep6_1536pix_16node" + + +srun \ +torchrun \ +--nnodes 16 \ +--nproc_per_node 8 \ +--rdzv_id $RANDOM \ +--rdzv_endpoint "${my_array[0]}:29500" \ +--rdzv_backend c10d \ +main.py \ +--output_dir ${EXP_DIR} \ +--with_box_refine --two_stage \ +--num_feature_levels 5 --num_queries 900 \ +--dim_feedforward 2048 --dropout 0.0 --cls_loss_coef 1.0 \ +--assign_first_stage --assign_second_stage \ +--epochs 6 --lr_drop 4 \ +--lr 5e-5 --lr_backbone 5e-5 --batch_size 1 \ +--backbone pev1 \ +--backbone_size Gwin384 \ +--backbone_init_values 0.1 \ +--backbone_tile_posemb True \ +--backbone_lrd 0.9 --backbone_layers 50 \ +--dataset_file objects365 \ +--coco_path /checkpoint/vision_encoder/public_data/objects365_v2 \ +--finetune /checkpoint/vision_encoder/d2_output/coco_sota/pretrain_spatial_Gwin384_o365ep12_1024pix_16node/checkpoint.pth \ +--lsj --lsj_img_size 1536 \ +--backbone_use_act_checkpoint --backbone_act_checkpoint_ratio 1.0 \ +--eval_per_epochs 1 \ +--save_per_epochs 1 \ +--auto_resume \ +"$@" diff --git a/perception_models/apps/detection/DETA_pe/scripts/pretrain_spatial_Gwin384_o365ep12_1024pix_16node.sh b/perception_models/apps/detection/DETA_pe/scripts/pretrain_spatial_Gwin384_o365ep12_1024pix_16node.sh new file mode 100644 index 0000000000000000000000000000000000000000..cdafa3321b321c2d5cdea4d80cb1c70d5ee0f9c7 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/scripts/pretrain_spatial_Gwin384_o365ep12_1024pix_16node.sh @@ -0,0 +1,58 @@ +#!/bin/bash + +#SBATCH --qos=vision_encoder +#SBATCH --account=vision_encoder +#SBATCH --job-name=det +#SBATCH --nodes=16 +#SBATCH --ntasks=16 +#SBATCH --gres=gpu:8 +#SBATCH --cpus-per-task=32 +#SBATCH --mem=0 +#SBATCH --output=/checkpoint/vision_encoder/d2_output/slurm_logs/coco_sota/pretrain_spatial_Gwin384_o365ep12_1024pix_16node/%j.out +#SBATCH --error=/checkpoint/vision_encoder/d2_output/slurm_logs/coco_sota/pretrain_spatial_Gwin384_o365ep12_1024pix_16node/%j.err +#SBATCH --time=96:00:00 + +module load cuda/12.1 +nodes=($(scontrol show hostnames $SLURM_JOB_NODELIST)) +nodes_array=($nodes) +head_node=${nodes_array[0]} +head_node_ip=$(srun --nodes=1 --ntasks=1 -w "$head_node" hostname --ip-address) + +read -ra my_array <<< $head_node_ip +export LOGLEVEL=INFO + +echo head_node_ip $head_node_ip +echo endpoint "${head_node_ip}:29500" + + +EXP_DIR="/checkpoint/vision_encoder/d2_output/coco_sota/pretrain_spatial_Gwin384_o365ep12_1024pix_16node" + +srun \ +torchrun \ +--nnodes 16 \ +--nproc_per_node 8 \ +--rdzv_id $RANDOM \ +--rdzv_endpoint "${my_array[0]}:29500" \ +--rdzv_backend c10d \ +main.py \ +--output_dir ${EXP_DIR} \ +--with_box_refine --two_stage \ +--num_feature_levels 5 --num_queries 900 \ +--dim_feedforward 2048 --dropout 0.0 --cls_loss_coef 1.0 \ +--assign_first_stage --assign_second_stage \ +--epochs 12 --lr_drop 10 \ +--lr_backbone 2e-4 \ +--backbone pev1 \ +--backbone_size Gwin384 \ +--backbone_path /checkpoint/vision_encoder/pev1/pe_spatial_G14_448_16patch384pix.pth \ +--backbone_init_values 0.1 \ +--backbone_tile_posemb True \ +--backbone_lrd 0.9 --backbone_layers 50 \ +--dataset_file objects365 \ +--coco_path /checkpoint/vision_encoder/public_data/objects365_v2 \ +--lsj --lsj_img_size 1024 \ +--backbone_use_act_checkpoint --backbone_act_checkpoint_ratio 1.0 \ +--eval_per_epochs 2 \ +--save_per_epochs 1 \ +--auto_resume \ +"$@" diff --git a/perception_models/apps/detection/DETA_pe/util/__init__.py b/perception_models/apps/detection/DETA_pe/util/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..4ebdc90b7f3ac2ed5a085066dcf20722b90cbc77 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/util/__init__.py @@ -0,0 +1,8 @@ +# ------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------ +# Modified from DETR (https://github.com/facebookresearch/detr) +# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved +# ------------------------------------------------------------------------ diff --git a/perception_models/apps/detection/DETA_pe/util/box_ops.py b/perception_models/apps/detection/DETA_pe/util/box_ops.py new file mode 100644 index 0000000000000000000000000000000000000000..ca29592f8077cba5f934bb364370ac09adca9e76 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/util/box_ops.py @@ -0,0 +1,96 @@ +# ------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------ +# Modified from DETR (https://github.com/facebookresearch/detr) +# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved +# ------------------------------------------------------------------------ + +""" +Utilities for bounding box manipulation and GIoU. +""" +import torch +from torchvision.ops.boxes import box_area + + +def box_cxcywh_to_xyxy(x): + x_c, y_c, w, h = x.unbind(-1) + b = [(x_c - 0.5 * w), (y_c - 0.5 * h), + (x_c + 0.5 * w), (y_c + 0.5 * h)] + return torch.stack(b, dim=-1) + + +def box_xyxy_to_cxcywh(x): + x0, y0, x1, y1 = x.unbind(-1) + b = [(x0 + x1) / 2, (y0 + y1) / 2, + (x1 - x0), (y1 - y0)] + return torch.stack(b, dim=-1) + + +# modified from torchvision to also return the union +def box_iou(boxes1, boxes2): + area1 = box_area(boxes1) + area2 = box_area(boxes2) + + lt = torch.max(boxes1[:, None, :2], boxes2[:, :2]) # [N,M,2] + rb = torch.min(boxes1[:, None, 2:], boxes2[:, 2:]) # [N,M,2] + + wh = (rb - lt).clamp(min=0) # [N,M,2] + inter = wh[:, :, 0] * wh[:, :, 1] # [N,M] + + union = area1[:, None] + area2 - inter + + iou = inter / union + return iou, union + + +def generalized_box_iou(boxes1, boxes2): + """ + Generalized IoU from https://giou.stanford.edu/ + + The boxes should be in [x0, y0, x1, y1] format + + Returns a [N, M] pairwise matrix, where N = len(boxes1) + and M = len(boxes2) + """ + # degenerate boxes gives inf / nan results + # so do an early check + assert (boxes1[:, 2:] >= boxes1[:, :2]).all() + assert (boxes2[:, 2:] >= boxes2[:, :2]).all() + iou, union = box_iou(boxes1, boxes2) + + lt = torch.min(boxes1[:, None, :2], boxes2[:, :2]) + rb = torch.max(boxes1[:, None, 2:], boxes2[:, 2:]) + + wh = (rb - lt).clamp(min=0) # [N,M,2] + area = wh[:, :, 0] * wh[:, :, 1] + + return iou - (area - union) / area + + +def masks_to_boxes(masks): + """Compute the bounding boxes around the provided masks + + The masks should be in format [N, H, W] where N is the number of masks, (H, W) are the spatial dimensions. + + Returns a [N, 4] tensors, with the boxes in xyxy format + """ + if masks.numel() == 0: + return torch.zeros((0, 4), device=masks.device) + + h, w = masks.shape[-2:] + + y = torch.arange(0, h, dtype=torch.float) + x = torch.arange(0, w, dtype=torch.float) + y, x = torch.meshgrid(y, x) + + x_mask = (masks * x.unsqueeze(0)) + x_max = x_mask.flatten(1).max(-1)[0] + x_min = x_mask.masked_fill(~(masks.bool()), 1e8).flatten(1).min(-1)[0] + + y_mask = (masks * y.unsqueeze(0)) + y_max = y_mask.flatten(1).max(-1)[0] + y_min = y_mask.masked_fill(~(masks.bool()), 1e8).flatten(1).min(-1)[0] + + return torch.stack([x_min, y_min, x_max, y_max], 1) diff --git a/perception_models/apps/detection/DETA_pe/util/ema.py b/perception_models/apps/detection/DETA_pe/util/ema.py new file mode 100644 index 0000000000000000000000000000000000000000..3c357b544c9b6722e18ab3969331d10970198521 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/util/ema.py @@ -0,0 +1,24 @@ +from collections import OrderedDict + +import torch + + +@torch.no_grad() +def update_ema(ema_model, model, decay=0.9999): + """ + Step the EMA model towards the current model. + """ + ema_params = OrderedDict(ema_model.named_parameters()) + model_params = OrderedDict(model.named_parameters()) + + for name, param in model_params.items(): + # TODO: Consider applying only to params that require_grad to avoid small numerical changes of pos_embed + ema_params[name].mul_(decay).add_(param.data, alpha=1 - decay) + + +def requires_grad(model, flag=True): + """ + Set requires_grad flag for all parameters in a model. + """ + for p in model.parameters(): + p.requires_grad = flag diff --git a/perception_models/apps/detection/DETA_pe/util/misc.py b/perception_models/apps/detection/DETA_pe/util/misc.py new file mode 100644 index 0000000000000000000000000000000000000000..85bc3f56560d5e1ec803361154b57fc329ae1bf7 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/util/misc.py @@ -0,0 +1,679 @@ +# ------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------ +# Modified from DETR (https://github.com/facebookresearch/detr) +# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved +# ------------------------------------------------------------------------ + +""" +Misc functions, including distributed helpers. + +Mostly copy-paste from torchvision references. +""" +import datetime +import os +import pickle +import subprocess +import time +from collections import defaultdict, deque +from typing import List, Optional + +import torch +import torch.distributed as dist +import torch.nn as nn + +# needed due to empty tensor bug in pytorch and torchvision 0.5 +import torchvision +from torch import Tensor + +if ( + float(torchvision.__version__.split(".")[0]) == 0 + and float(torchvision.__version__.split(".")[1]) < 5 +): + import math + + from torchvision.ops.misc import _NewEmptyTensorOp + + def _check_size_scale_factor(dim, size, scale_factor): + # type: (int, Optional[List[int]], Optional[float]) -> None + if size is None and scale_factor is None: + raise ValueError("either size or scale_factor should be defined") + if size is not None and scale_factor is not None: + raise ValueError("only one of size or scale_factor should be defined") + if not (scale_factor is not None and len(scale_factor) != dim): + raise ValueError( + "scale_factor shape must match input shape. " + "Input is {}D, scale_factor size is {}".format(dim, len(scale_factor)) + ) + + def _output_size(dim, input, size, scale_factor): + # type: (int, Tensor, Optional[List[int]], Optional[float]) -> List[int] + assert dim == 2 + _check_size_scale_factor(dim, size, scale_factor) + if size is not None: + return size + # if dim is not 2 or scale_factor is iterable use _ntuple instead of concat + assert scale_factor is not None and isinstance(scale_factor, (int, float)) + scale_factors = [scale_factor, scale_factor] + # math.floor might return float in py2.7 + return [ + int(math.floor(input.size(i + 2) * scale_factors[i])) for i in range(dim) + ] + +elif ( + float(torchvision.__version__.split(".")[0]) == 0 + and float(torchvision.__version__.split(".")[1]) < 7 +): + from torchvision.ops import _new_empty_tensor + from torchvision.ops.misc import _output_size + + +class SmoothedValue(object): + """Track a series of values and provide access to smoothed values over a + window or the global series average. + """ + + def __init__(self, window_size=20, fmt=None): + if fmt is None: + fmt = "{median:.4f} ({global_avg:.4f})" + self.deque = deque(maxlen=window_size) + self.total = 0.0 + self.count = 0 + self.fmt = fmt + + def update(self, value, n=1): + self.deque.append(value) + self.count += n + self.total += value * n + + def synchronize_between_processes(self): + """ + Warning: does not synchronize the deque! + """ + if not is_dist_avail_and_initialized(): + return + t = torch.tensor([self.count, self.total], dtype=torch.float64, device="cuda") + dist.barrier() + dist.all_reduce(t) + t = t.tolist() + self.count = int(t[0]) + self.total = t[1] + + @property + def median(self): + d = torch.tensor(list(self.deque)) + return d.median().item() + + @property + def avg(self): + d = torch.tensor(list(self.deque), dtype=torch.float32) + return d.mean().item() + + @property + def global_avg(self): + return self.total / self.count + + @property + def max(self): + return max(self.deque) + + @property + def value(self): + return self.deque[-1] + + def __str__(self): + return self.fmt.format( + median=self.median, + avg=self.avg, + global_avg=self.global_avg, + max=self.max, + value=self.value, + ) + + +def all_gather(data): + """ + Run all_gather on arbitrary picklable data (not necessarily tensors) + Args: + data: any picklable object + Returns: + list[data]: list of data gathered from each rank + """ + world_size = get_world_size() + if world_size == 1: + return [data] + + # serialized to a Tensor + buffer = pickle.dumps(data) + storage = torch.ByteStorage.from_buffer(buffer) + tensor = torch.ByteTensor(storage).to("cuda") + + # obtain Tensor size of each rank + local_size = torch.tensor([tensor.numel()], device="cuda") + size_list = [torch.tensor([0], device="cuda") for _ in range(world_size)] + dist.all_gather(size_list, local_size) + size_list = [int(size.item()) for size in size_list] + max_size = max(size_list) + + # receiving Tensor from all ranks + # we pad the tensor because torch all_gather does not support + # gathering tensors of different shapes + tensor_list = [] + for _ in size_list: + tensor_list.append(torch.empty((max_size,), dtype=torch.uint8, device="cuda")) + if local_size != max_size: + padding = torch.empty( + size=(max_size - local_size,), dtype=torch.uint8, device="cuda" + ) + tensor = torch.cat((tensor, padding), dim=0) + dist.all_gather(tensor_list, tensor) + + data_list = [] + for size, tensor in zip(size_list, tensor_list): + buffer = tensor.cpu().numpy().tobytes()[:size] + data_list.append(pickle.loads(buffer)) + + return data_list + + +def reduce_dict(input_dict, average=True): + """ + Args: + input_dict (dict): all the values will be reduced + average (bool): whether to do average or sum + Reduce the values in the dictionary from all processes so that all processes + have the averaged results. Returns a dict with the same fields as + input_dict, after reduction. + """ + world_size = get_world_size() + if world_size < 2: + return input_dict + with torch.no_grad(): + names = [] + values = [] + # sort the keys so that they are consistent across processes + for k in sorted(input_dict.keys()): + names.append(k) + values.append(input_dict[k]) + values = torch.stack(values, dim=0) + dist.all_reduce(values) + if average: + values /= world_size + reduced_dict = {k: v for k, v in zip(names, values)} + return reduced_dict + + +class MetricLogger(object): + def __init__(self, delimiter="\t"): + self.meters = defaultdict(SmoothedValue) + self.delimiter = delimiter + + def update(self, **kwargs): + for k, v in kwargs.items(): + if isinstance(v, torch.Tensor): + v = v.item() + assert isinstance(v, (float, int)) + self.meters[k].update(v) + + def __getattr__(self, attr): + if attr in self.meters: + return self.meters[attr] + if attr in self.__dict__: + return self.__dict__[attr] + raise AttributeError( + "'{}' object has no attribute '{}'".format(type(self).__name__, attr) + ) + + def __str__(self): + loss_str = [] + for name, meter in self.meters.items(): + loss_str.append("{}: {}".format(name, str(meter))) + return self.delimiter.join(loss_str) + + def synchronize_between_processes(self): + for meter in self.meters.values(): + meter.synchronize_between_processes() + + def add_meter(self, name, meter): + self.meters[name] = meter + + def log_every(self, iterable, print_freq, header=None): + i = 0 + if not header: + header = "" + start_time = time.time() + end = time.time() + iter_time = SmoothedValue(fmt="{avg:.4f}") + data_time = SmoothedValue(fmt="{avg:.4f}") + space_fmt = ":" + str(len(str(len(iterable)))) + "d" + if torch.cuda.is_available(): + log_msg = self.delimiter.join( + [ + header, + "[{0" + space_fmt + "}/{1}]", + "eta: {eta}", + "{meters}", + "time: {time}", + "data: {data}", + "max mem: {memory:.0f}", + ] + ) + else: + log_msg = self.delimiter.join( + [ + header, + "[{0" + space_fmt + "}/{1}]", + "eta: {eta}", + "{meters}", + "time: {time}", + "data: {data}", + ] + ) + MB = 1024.0 * 1024.0 + for obj in iterable: + data_time.update(time.time() - end) + yield obj + iter_time.update(time.time() - end) + if i % print_freq == 0 or i == len(iterable) - 1: + eta_seconds = iter_time.global_avg * (len(iterable) - i) + eta_string = str(datetime.timedelta(seconds=int(eta_seconds))) + if torch.cuda.is_available(): + print( + log_msg.format( + i, + len(iterable), + eta=eta_string, + meters=str(self), + time=str(iter_time), + data=str(data_time), + memory=torch.cuda.max_memory_allocated() / MB, + ) + ) + else: + print( + log_msg.format( + i, + len(iterable), + eta=eta_string, + meters=str(self), + time=str(iter_time), + data=str(data_time), + ) + ) + i += 1 + end = time.time() + total_time = time.time() - start_time + total_time_str = str(datetime.timedelta(seconds=int(total_time))) + print( + "{} Total time: {} ({:.4f} s / it)".format( + header, total_time_str, total_time / len(iterable) + ) + ) + + +def get_sha(): + cwd = os.path.dirname(os.path.abspath(__file__)) + + def _run(command): + return subprocess.check_output(command, cwd=cwd).decode("ascii").strip() + + sha = "N/A" + diff = "clean" + branch = "N/A" + try: + sha = _run(["git", "rev-parse", "HEAD"]) + subprocess.check_output(["git", "diff"], cwd=cwd) + diff = _run(["git", "diff-index", "HEAD"]) + diff = "has uncommited changes" if diff else "clean" + branch = _run(["git", "rev-parse", "--abbrev-ref", "HEAD"]) + except Exception: + pass + message = f"sha: {sha}, status: {diff}, branch: {branch}" + return message + + +def collate_fn(batch): + batch = list(zip(*batch)) + batch[0] = nested_tensor_from_tensor_list(batch[0]) + return tuple(batch) + + +class CollatorLSJMultiscale: + def __init__(self, lsj_img_size=1024, tta=False): + self.lsj_img_size_set = [1120, 1344, 1568, 1680, 1792] #TODO: make it configurable + self.tta = tta + + def nested_tensor_from_tensor_list_lsj(self, tensor_list: List[Tensor]): + # TODO make this more general + if tensor_list[0].ndim == 3: + # TODO make it support different-sized images + max_size = _max_by_axis([list(img.shape) for img in tensor_list]) + # min_size = tuple(min(s) for s in zip(*[img.shape for img in tensor_list])) + # batch_shape = [len(tensor_list)] + max_size + height, width = max_size[-2], max_size[-1] + max_size_len = height if height >= width else width + + lsj_img_size = self.lsj_img_size_set[0] + for i in range(len(self.lsj_img_size_set)): + if max_size_len <= self.lsj_img_size_set[i]: + lsj_img_size = self.lsj_img_size_set[i] + break + batch_shape = [len(tensor_list)] + [ + max_size[0], + lsj_img_size, + lsj_img_size, + ] + b, c, h, w = batch_shape + dtype = tensor_list[0].dtype + device = tensor_list[0].device + tensor = torch.zeros(batch_shape, dtype=dtype, device=device) + mask = torch.ones((b, h, w), dtype=torch.bool, device=device) + for img, pad_img, m in zip(tensor_list, tensor, mask): + pad_img[: img.shape[0], : img.shape[1], : img.shape[2]].copy_(img) + m[: img.shape[1], : img.shape[2]] = False + else: + raise ValueError("not supported") + + return NestedTensor(tensor, mask) + + def __call__(self, batch): + batch = list(zip(*batch)) + batch[0] = self.nested_tensor_from_tensor_list_lsj(batch[0]) + return tuple(batch) + + +class CollatorLSJ: + def __init__(self, lsj_img_size=1024, tta=False): + self.lsj_img_size = lsj_img_size + self.tta = tta + + def nested_tensor_from_tensor_list_lsj(self, tensor_list: List[Tensor]): + if self.tta: + # num_batch = len(tensor_list) + # num_channels = len(tensor_list[0]) * 3 + assert len(tensor_list) == 1, "only support one image in tta" + batch_shape = [ + len(tensor_list), + len(tensor_list[0]) * 3, + self.lsj_img_size, + self.lsj_img_size, + ] + b, c, h, w = batch_shape + dtype = tensor_list[0][0].dtype + device = tensor_list[0][0].device + tensor = torch.zeros(batch_shape, dtype=dtype, device=device) + mask = torch.ones( + (b, len(tensor_list[0]), h, w), dtype=torch.bool, device=device + ) + for scale_idx in range(len(tensor_list[0])): + img = tensor_list[0][scale_idx] + pad_img = tensor[0][scale_idx * 3 : (scale_idx + 1) * 3] + pad_img[: img.shape[0], : img.shape[1], : img.shape[2]].copy_(img) + pad_mask = mask[0][scale_idx] + pad_mask[: img.shape[1], : img.shape[2]] = False + + # breakpoint() + # for img, pad_img, m in zip(tensor_list, tensor, mask): + # pad_img[: img.shape[0], : img.shape[1], : img.shape[2]].copy_(img) + # m[: img.shape[1], : img.shape[2]] = False + + return NestedTensor(tensor, mask) + + # TODO make this more general + if tensor_list[0].ndim == 3 or tensor_list[0].ndim == 6: + # TODO make it support different-sized images + orig_sizes = [list(img.shape) for img in tensor_list] + max_size = _max_by_axis(orig_sizes) + assert ( + max(max_size[-2:]) <= self.lsj_img_size + ), f"orig_sizes: {orig_sizes}, max_size: {max_size}, lsj_img_size: {self.lsj_img_size}" + # batch_shape = [len(tensor_list)] + max_size + batch_shape = [len(tensor_list)] + [ + max_size[0], + self.lsj_img_size, + self.lsj_img_size, + ] + b, c, h, w = batch_shape + dtype = tensor_list[0].dtype + device = tensor_list[0].device + tensor = torch.zeros(batch_shape, dtype=dtype, device=device) + mask = torch.ones((b, h, w), dtype=torch.bool, device=device) + for img, pad_img, m in zip(tensor_list, tensor, mask): + pad_img[: img.shape[0], : img.shape[1], : img.shape[2]].copy_(img) + m[: img.shape[1], : img.shape[2]] = False + else: + raise ValueError("not supported") + return NestedTensor(tensor, mask) + + def __call__(self, batch): + batch = list(zip(*batch)) + batch[0] = self.nested_tensor_from_tensor_list_lsj(batch[0]) + return tuple(batch) + + +def _max_by_axis(the_list): + # type: (List[List[int]]) -> List[int] + maxes = the_list[0] + for sublist in the_list[1:]: + for index, item in enumerate(sublist): + maxes[index] = max(maxes[index], item) + return maxes + + +def nested_tensor_from_tensor_list(tensor_list: List[Tensor]): + # TODO make this more general + if tensor_list[0].ndim == 3: + # TODO make it support different-sized images + max_size = _max_by_axis([list(img.shape) for img in tensor_list]) + # min_size = tuple(min(s) for s in zip(*[img.shape for img in tensor_list])) + batch_shape = [len(tensor_list)] + max_size + b, c, h, w = batch_shape + dtype = tensor_list[0].dtype + device = tensor_list[0].device + tensor = torch.zeros(batch_shape, dtype=dtype, device=device) + mask = torch.ones((b, h, w), dtype=torch.bool, device=device) + for img, pad_img, m in zip(tensor_list, tensor, mask): + pad_img[: img.shape[0], : img.shape[1], : img.shape[2]].copy_(img) + m[: img.shape[1], : img.shape[2]] = False + else: + raise ValueError("not supported") + return NestedTensor(tensor, mask) + + +class NestedTensor(object): + def __init__(self, tensors, mask: Optional[Tensor]): + self.tensors = tensors + self.mask = mask + + def to(self, device, non_blocking=False): + # type: (Device) -> NestedTensor # noqa + cast_tensor = self.tensors.to(device, non_blocking=non_blocking) + mask = self.mask + if mask is not None: + assert mask is not None + cast_mask = mask.to(device, non_blocking=non_blocking) + else: + cast_mask = None + return NestedTensor(cast_tensor, cast_mask) + + def record_stream(self, *args, **kwargs): + self.tensors.record_stream(*args, **kwargs) + if self.mask is not None: + self.mask.record_stream(*args, **kwargs) + + def decompose(self): + return self.tensors, self.mask + + def __repr__(self): + return str(self.tensors) + + +def setup_for_distributed(is_master): + """ + This function disables printing when not in master process + """ + import builtins as __builtin__ + + builtin_print = __builtin__.print + + def print(*args, **kwargs): + force = kwargs.pop("force", False) + if is_master or force: + builtin_print(*args, **kwargs) + + __builtin__.print = print + + +def is_dist_avail_and_initialized(): + if not dist.is_available(): + return False + if not dist.is_initialized(): + return False + return True + + +def get_world_size(): + if not is_dist_avail_and_initialized(): + return 1 + return dist.get_world_size() + + +def get_rank(): + if not is_dist_avail_and_initialized(): + return 0 + return dist.get_rank() + + +def get_local_size(): + if not is_dist_avail_and_initialized(): + return 1 + return int(os.environ["LOCAL_SIZE"]) + + +def get_local_rank(): + if not is_dist_avail_and_initialized(): + return 0 + return int(os.environ["LOCAL_RANK"]) + + +def is_main_process(): + return get_rank() == 0 + + +def save_on_master(*args, **kwargs): + if is_main_process(): + torch.save(*args, **kwargs) + + +def init_distributed_mode(args): + if "RANK" in os.environ and "WORLD_SIZE" in os.environ: + args.rank = int(os.environ["RANK"]) + args.world_size = int(os.environ["WORLD_SIZE"]) + args.gpu = int(os.environ["LOCAL_RANK"]) + args.dist_url = "env://" + os.environ["LOCAL_SIZE"] = str(torch.cuda.device_count()) + elif "SLURM_PROCID" in os.environ: + proc_id = int(os.environ["SLURM_PROCID"]) + ntasks = int(os.environ["SLURM_NTASKS"]) + node_list = os.environ["SLURM_NODELIST"] + num_gpus = torch.cuda.device_count() + addr = subprocess.getoutput( + "scontrol show hostname {} | head -n1".format(node_list) + ) + os.environ["MASTER_PORT"] = os.environ.get("MASTER_PORT", "29500") + os.environ["MASTER_ADDR"] = addr + os.environ["WORLD_SIZE"] = str(ntasks) + os.environ["RANK"] = str(proc_id) + os.environ["LOCAL_RANK"] = str(proc_id % num_gpus) + os.environ["LOCAL_SIZE"] = str(num_gpus) + args.dist_url = "env://" + args.world_size = ntasks + args.rank = proc_id + args.gpu = proc_id % num_gpus + else: + print("Not using distributed mode") + args.distributed = False + return + + args.distributed = True + + torch.cuda.set_device(args.gpu) + args.dist_backend = "nccl" + print( + "| distributed init (rank {}): {}".format(args.rank, args.dist_url), flush=True + ) + torch.distributed.init_process_group( + backend=args.dist_backend, + init_method=args.dist_url, + world_size=args.world_size, + rank=args.rank, + ) + torch.distributed.barrier() + setup_for_distributed(args.rank == 0) + + +@torch.no_grad() +def accuracy(output, target, topk=(1,)): + """Computes the precision@k for the specified values of k""" + if target.numel() == 0: + return [torch.zeros([], device=output.device)] + maxk = max(topk) + batch_size = target.size(0) + + _, pred = output.topk(maxk, 1, True, True) + pred = pred.t() + correct = pred.eq(target.view(1, -1).expand_as(pred)) + + res = [] + for k in topk: + correct_k = correct[:k].view(-1).float().sum(0) + res.append(correct_k.mul_(100.0 / batch_size)) + return res + + +def interpolate( + input, size=None, scale_factor=None, mode="nearest", align_corners=None +): + # type: (Tensor, Optional[List[int]], Optional[float], str, Optional[bool]) -> Tensor + """ + Equivalent to nn.functional.interpolate, but with support for empty batch sizes. + This will eventually be supported natively by PyTorch, and this + class can go away. + """ + if float(torchvision.__version__[:3]) < 0.7: + if input.numel() > 0: + return torch.nn.functional.interpolate( + input, size, scale_factor, mode, align_corners + ) + + output_shape = _output_size(2, input, size, scale_factor) + output_shape = list(input.shape[:-2]) + list(output_shape) + if float(torchvision.__version__[:3]) < 0.5: + return _NewEmptyTensorOp.apply(input, output_shape) + return _new_empty_tensor(input, output_shape) + else: + return torchvision.ops.misc.interpolate( + input, size, scale_factor, mode, align_corners + ) + + +def get_total_grad_norm(parameters, norm_type=2): + parameters = list(filter(lambda p: p.grad is not None, parameters)) + norm_type = float(norm_type) + device = parameters[0].grad.device + total_norm = torch.norm( + torch.stack( + [torch.norm(p.grad.detach(), norm_type).to(device) for p in parameters] + ), + norm_type, + ) + return total_norm + + +def inverse_sigmoid(x, eps=1e-5): + x = x.clamp(min=0, max=1) + x1 = x.clamp(min=eps) + x2 = (1 - x).clamp(min=eps) + return torch.log(x1 / x2) diff --git a/perception_models/apps/detection/DETA_pe/util/plot_utils.py b/perception_models/apps/detection/DETA_pe/util/plot_utils.py new file mode 100644 index 0000000000000000000000000000000000000000..759f34d252493fd93187ea3cf2ab0d63a3e2b280 --- /dev/null +++ b/perception_models/apps/detection/DETA_pe/util/plot_utils.py @@ -0,0 +1,111 @@ +# ------------------------------------------------------------------------ +# Deformable DETR +# Copyright (c) 2020 SenseTime. All Rights Reserved. +# Licensed under the Apache License, Version 2.0 [see LICENSE for details] +# ------------------------------------------------------------------------ +# Modified from DETR (https://github.com/facebookresearch/detr) +# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved +# ------------------------------------------------------------------------ + +""" +Plotting utilities to visualize training logs. +""" +import torch +import pandas as pd +import seaborn as sns +import matplotlib.pyplot as plt + +from pathlib import Path, PurePath + + +def plot_logs(logs, fields=('class_error', 'loss_bbox_unscaled', 'mAP'), ewm_col=0, log_name='log.txt'): + ''' + Function to plot specific fields from training log(s). Plots both training and test results. + + :: Inputs - logs = list containing Path objects, each pointing to individual dir with a log file + - fields = which results to plot from each log file - plots both training and test for each field. + - ewm_col = optional, which column to use as the exponential weighted smoothing of the plots + - log_name = optional, name of log file if different than default 'log.txt'. + + :: Outputs - matplotlib plots of results in fields, color coded for each log file. + - solid lines are training results, dashed lines are test results. + + ''' + func_name = "plot_utils.py::plot_logs" + + # verify logs is a list of Paths (list[Paths]) or single Pathlib object Path, + # convert single Path to list to avoid 'not iterable' error + + if not isinstance(logs, list): + if isinstance(logs, PurePath): + logs = [logs] + print(f"{func_name} info: logs param expects a list argument, converted to list[Path].") + else: + raise ValueError(f"{func_name} - invalid argument for logs parameter.\n \ + Expect list[Path] or single Path obj, received {type(logs)}") + + # verify valid dir(s) and that every item in list is Path object + for i, dir in enumerate(logs): + if not isinstance(dir, PurePath): + raise ValueError(f"{func_name} - non-Path object in logs argument of {type(dir)}: \n{dir}") + if dir.exists(): + continue + raise ValueError(f"{func_name} - invalid directory in logs argument:\n{dir}") + + # load log file(s) and plot + dfs = [pd.read_json(Path(p) / log_name, lines=True) for p in logs] + + fig, axs = plt.subplots(ncols=len(fields), figsize=(16, 5)) + + for df, color in zip(dfs, sns.color_palette(n_colors=len(logs))): + for j, field in enumerate(fields): + if field == 'mAP': + coco_eval = pd.DataFrame(pd.np.stack(df.test_coco_eval.dropna().values)[:, 1]).ewm(com=ewm_col).mean() + axs[j].plot(coco_eval, c=color) + else: + df.interpolate().ewm(com=ewm_col).mean().plot( + y=[f'train_{field}', f'test_{field}'], + ax=axs[j], + color=[color] * 2, + style=['-', '--'] + ) + for ax, field in zip(axs, fields): + ax.legend([Path(p).name for p in logs]) + ax.set_title(field) + + +def plot_precision_recall(files, naming_scheme='iter'): + if naming_scheme == 'exp_id': + # name becomes exp_id + names = [f.parts[-3] for f in files] + elif naming_scheme == 'iter': + names = [f.stem for f in files] + else: + raise ValueError(f'not supported {naming_scheme}') + fig, axs = plt.subplots(ncols=2, figsize=(16, 5)) + for f, color, name in zip(files, sns.color_palette("Blues", n_colors=len(files)), names): + data = torch.load(f) + # precision is n_iou, n_points, n_cat, n_area, max_det + precision = data['precision'] + recall = data['params'].recThrs + scores = data['scores'] + # take precision for all classes, all areas and 100 detections + precision = precision[0, :, :, 0, -1].mean(1) + scores = scores[0, :, :, 0, -1].mean(1) + prec = precision.mean() + rec = data['recall'][0, :, 0, -1].mean() + print(f'{naming_scheme} {name}: mAP@50={prec * 100: 05.1f}, ' + + f'score={scores.mean():0.3f}, ' + + f'f1={2 * prec * rec / (prec + rec + 1e-8):0.3f}' + ) + axs[0].plot(recall, precision, c=color) + axs[1].plot(recall, scores, c=color) + + axs[0].set_title('Precision / Recall') + axs[0].legend(names) + axs[1].set_title('Scores / Recall') + axs[1].legend(names) + return fig, axs + + + diff --git a/perception_models/apps/detection/INSTALL.md b/perception_models/apps/detection/INSTALL.md new file mode 100644 index 0000000000000000000000000000000000000000..cb4dd07cde075ecee1f69963b49c09561ac5b284 --- /dev/null +++ b/perception_models/apps/detection/INSTALL.md @@ -0,0 +1,19 @@ +## Installation +Follow [Detectron2 installation instructions](https://detectron2.readthedocs.io/tutorials/install.html) + +## Dataset +Prepare COCO and LVIS datasets + +``` +$DETECTRON2_DATASETS/ + coco/ + train2017/ + val2017/ + annotations/ + instances_train2017.json + instances_val2017.json + lvis/ + lvis_v1_train.json + lvis_v1_val.json +``` + diff --git a/perception_models/apps/detection/README.md b/perception_models/apps/detection/README.md new file mode 100644 index 0000000000000000000000000000000000000000..fee286cddf9258b06dcbd4b799b335f54c26fd7d --- /dev/null +++ b/perception_models/apps/detection/README.md @@ -0,0 +1,106 @@ +# Object Detection with PE + +## Getting started + +Please refer to [INSTALL.md](INSTALL.md) for installation and dataset preparation instructions. + +## Results and Fine-tuned Models + + +### LVIS + + + + + + + + + + + + + + + + + + + + + + + + +
detectorvision encoderbox
AP		mask
AP		download
Mask R-CNNPE core G51.947.9model
Mask R-CNNPE spatial G54.249.3model
+ + +### COCO + + + + + + + + + + + + + + + + + + + + + + + + +
detectorvision encoderbox
AP		mask
AP		download
Mask R-CNNPE core G57.049.8model
Mask R-CNNPE spatial G57.850.3model
+ + +### Training +By default, we use 64 GPUs in slurm training, for example + +``` +sbatch scripts/coco/train_mask_rcnn_PEspatial_G_coco36ep.sh +``` + +### Evaluation +Evaluation is running locally +``` +bash scripts/evaluate_local.sh --config-file projects/ViTDet/configs/COCO/mask_rcnn_PEspatial_G_coco36ep.py train.output_dir="/path/to/output_dir" train.init_checkpoint="/path/to/mask_rcnn_PEspatial_G_coco36ep.pth" +``` + + +## SOTA COCO Object Detection + + + + + + + + + + + + + + + + + + +
detectorvision encoderbox
AP		box(TTA)
AP		download
DETAPE spatial G 65.2 66.0 model
+ +More details are in [DETA_pe](DETA_pe) + + +## Acknowledgment + +This code is built using [detectron2](https://github.com/facebookresearch/detectron2) and [DETA](https://github.com/jozhang97/DETA). diff --git a/perception_models/apps/detection/detectron2_pe/__init__.py b/perception_models/apps/detection/detectron2_pe/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..283d9e0cc7512c4a46792daee78a8ededd7bd56a --- /dev/null +++ b/perception_models/apps/detection/detectron2_pe/__init__.py @@ -0,0 +1 @@ +from . import modeling diff --git a/perception_models/apps/detection/detectron2_pe/checkpoint/__init__.py b/perception_models/apps/detection/detectron2_pe/checkpoint/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..b43ad2512a8206e0f4b4fe133605d457b9e384d2 --- /dev/null +++ b/perception_models/apps/detection/detectron2_pe/checkpoint/__init__.py @@ -0,0 +1,3 @@ +from .detection_checkpoint import DetectionCheckpointer + +__all__ = ["DetectionCheckpointer"] diff --git a/perception_models/apps/detection/detectron2_pe/checkpoint/detection_checkpoint.py b/perception_models/apps/detection/detectron2_pe/checkpoint/detection_checkpoint.py new file mode 100644 index 0000000000000000000000000000000000000000..56fd16f106f9187f9359a9b880f830a532633848 --- /dev/null +++ b/perception_models/apps/detection/detectron2_pe/checkpoint/detection_checkpoint.py @@ -0,0 +1,152 @@ +# Copyright (c) Facebook, Inc. and its affiliates. +import logging +import os +import pickle +from urllib.parse import parse_qs, urlparse + +import detectron2.utils.comm as comm +import torch +from detectron2.checkpoint.c2_model_loading import align_and_update_state_dicts +from detectron2.utils.file_io import PathManager +from fvcore.common.checkpoint import Checkpointer +from torch.nn.parallel import DistributedDataParallel + + +class DetectionCheckpointer(Checkpointer): + """ + Same as :class:`Checkpointer`, but is able to: + 1. handle models in detectron & detectron2 model zoo, and apply conversions for legacy models. + 2. correctly load checkpoints that are only available on the master worker + """ + + def __init__(self, model, save_dir="", *, save_to_disk=None, **checkpointables): + is_main_process = comm.is_main_process() + super().__init__( + model, + save_dir, + save_to_disk=is_main_process if save_to_disk is None else save_to_disk, + **checkpointables, + ) + self.path_manager = PathManager + self._parsed_url_during_load = None + + def load(self, path, *args, **kwargs): + assert self._parsed_url_during_load is None + need_sync = False + logger = logging.getLogger(__name__) + logger.info("[DetectionCheckpointer] Loading from {} ...".format(path)) + + if path and isinstance(self.model, DistributedDataParallel): + path = self.path_manager.get_local_path(path) + has_file = os.path.isfile(path) + all_has_file = comm.all_gather(has_file) + if not all_has_file[0]: + raise OSError(f"File {path} not found on main worker.") + if not all(all_has_file): + logger.warning( + f"Not all workers can read checkpoint {path}. " + "Training may fail to fully resume." + ) + # TODO: broadcast the checkpoint file contents from main + # worker, and load from it instead. + need_sync = True + if not has_file: + path = None # don't load if not readable + + # if path: + # parsed_url = urlparse(path) + # self._parsed_url_during_load = parsed_url + # path = parsed_url._replace(query="").geturl() # remove query from filename + # path = self.path_manager.get_local_path(path) + ret = super().load(path, *args, **kwargs) + + if need_sync: + logger.info("Broadcasting model states from main worker ...") + self.model._sync_params_and_buffers() + self._parsed_url_during_load = None # reset to None + return ret + + def _load_file(self, filename): + if filename.endswith(".pkl"): + with PathManager.open(filename, "rb") as f: + data = pickle.load(f, encoding="latin1") + if "model" in data and "__author__" in data: + # file is in Detectron2 model zoo format + self.logger.info("Reading a file from '{}'".format(data["__author__"])) + return data + else: + # assume file is from Caffe2 / Detectron1 model zoo + if "blobs" in data: + # Detection models have "blobs", but ImageNet models don't + data = data["blobs"] + data = {k: v for k, v in data.items() if not k.endswith("_momentum")} + return { + "model": data, + "__author__": "Caffe2", + "matching_heuristics": True, + } + elif filename.endswith(".pyth"): + # assume file is from pycls; no one else seems to use the ".pyth" extension + with PathManager.open(filename, "rb") as f: + data = torch.load(f) + assert ( + "model_state" in data + ), f"Cannot load .pyth file {filename}; pycls checkpoints must contain 'model_state'." + model_state = { + k: v + for k, v in data["model_state"].items() + if not k.endswith("num_batches_tracked") + } + return { + "model": model_state, + "__author__": "pycls", + "matching_heuristics": True, + } + + loaded = self._torch_load(filename) + if "model" not in loaded: + loaded = {"model": loaded} + + # assert self._parsed_url_during_load is not None, "`_load_file` must be called inside `load`" + # parsed_url = self._parsed_url_during_load + # queries = parse_qs(parsed_url.query) + # if queries.pop("matching_heuristics", "False") == ["True"]: + # loaded["matching_heuristics"] = True + # if len(queries) > 0: + # raise ValueError( + # f"Unsupported query remaining: f{queries}, orginal filename: {parsed_url.geturl()}" + # ) + loaded["matching_heuristics"] = True + return loaded + + def _torch_load(self, f): + return super()._load_file(f) + + def _load_model(self, checkpoint): + if checkpoint.get("matching_heuristics", False): + self._convert_ndarray_to_tensor(checkpoint["model"]) + # convert weights by name-matching heuristics + checkpoint["model"] = align_and_update_state_dicts( + self.model.state_dict(), + checkpoint["model"], + c2_conversion=checkpoint.get("__author__", None) == "Caffe2", + ) + # for non-caffe2 models, use standard ways to load it + incompatible = super()._load_model(checkpoint) + + model_buffers = dict(self.model.named_buffers(recurse=False)) + for k in ["pixel_mean", "pixel_std"]: + # Ignore missing key message about pixel_mean/std. + # Though they may be missing in old checkpoints, they will be correctly + # initialized from config anyway. + if k in model_buffers: + try: + incompatible.missing_keys.remove(k) + except ValueError: + pass + for k in incompatible.unexpected_keys[:]: + # Ignore unexpected keys about cell anchors. They exist in old checkpoints + # but now they are non-persistent buffers and will not be in new checkpoints. + if "anchor_generator.cell_anchors" in k: + incompatible.unexpected_keys.remove(k) + return incompatible diff --git a/perception_models/apps/detection/detectron2_pe/modeling/__init__.py b/perception_models/apps/detection/detectron2_pe/modeling/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..9ca54fc5603de935af1b082f7704cd084ea56bc8 --- /dev/null +++ b/perception_models/apps/detection/detectron2_pe/modeling/__init__.py @@ -0,0 +1 @@ +from .backbone import PEv1_det, get_vit_lr_decay_rate_pev1 diff --git a/perception_models/apps/detection/detectron2_pe/modeling/backbone/__init__.py b/perception_models/apps/detection/detectron2_pe/modeling/backbone/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..d1626c83bbb6e0ca66796e008a7a3537594e6c57 --- /dev/null +++ b/perception_models/apps/detection/detectron2_pe/modeling/backbone/__init__.py @@ -0,0 +1 @@ +from .pev1_det import PEv1_det, get_vit_lr_decay_rate_pev1 diff --git a/perception_models/apps/detection/detectron2_pe/modeling/backbone/pev1_det.py b/perception_models/apps/detection/detectron2_pe/modeling/backbone/pev1_det.py new file mode 100644 index 0000000000000000000000000000000000000000..353048f081ceb84e38776f250828f841743af7a6 --- /dev/null +++ b/perception_models/apps/detection/detectron2_pe/modeling/backbone/pev1_det.py @@ -0,0 +1,579 @@ +import logging +import math +from collections import OrderedDict +from functools import partial +from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple, Union + +import fvcore.nn.weight_init as weight_init +import torch +import torch.nn.functional as F +from detectron2.layers import CNNBlockBase, Conv2d, get_norm +from detectron2.modeling.backbone.backbone import Backbone +from detectron2.modeling.backbone.fpn import \ + _assert_strides_are_log2_contiguous +from detectron2.modeling.backbone.utils import PatchEmbed # get_abs_pos, +from detectron2.modeling.backbone.utils import (add_decomposed_rel_pos, + window_partition, + window_unpartition) +from einops import rearrange, repeat +from torch import broadcast_tensors, einsum, nn +from torch.nn.parameter import Parameter +from torch.utils.checkpoint import checkpoint + +logger = logging.getLogger(__name__) + + +__all__ = ["PEv1_det"] + + +def get_abs_pos(abs_pos, has_cls_token, hw, tile=False): + h, w = hw + if has_cls_token: + abs_pos = abs_pos[:, 1:] + xy_num = abs_pos.shape[1] + size = int(math.sqrt(xy_num)) + assert size * size == xy_num + + if size != h or size != w: + if tile == True: + new_abs_pos = abs_pos.reshape(1, size, size, -1).tile( + [1, h // size + 1, w // size + 1, 1] + )[:, :h, :w, :] + + return new_abs_pos + else: + new_abs_pos = F.interpolate( + abs_pos.reshape(1, size, size, -1).permute(0, 3, 1, 2), + size=(h, w), + mode="bicubic", + align_corners=False, + ) + return new_abs_pos.permute(0, 2, 3, 1) + else: + return abs_pos.reshape(1, h, w, -1) + + +# broadcat, as tortoise-tts was using it +def broadcat(tensors, dim=-1): + broadcasted_tensors = broadcast_tensors(*tensors) + return torch.cat(broadcasted_tensors, dim=dim) + + +# rotary embedding helper functions +def rotate_half(x): + x = rearrange(x, "... (d r) -> ... d r", r=2) + x1, x2 = x.unbind(dim=-1) + x = torch.stack((-x2, x1), dim=-1) + return rearrange(x, "... d r -> ... (d r)") + + +class VisionRotaryEmbeddingFast(nn.Module): + def __init__( + self, + dim, + pt_seq_len=16, + ft_seq_len=None, + custom_freqs=None, + freqs_for="lang", + theta=10000, + max_freq=10, + num_freqs=1, + ): + super().__init__() + if custom_freqs: + freqs = custom_freqs + elif freqs_for == "lang": + freqs = 1.0 / ( + theta ** (torch.arange(0, dim, 2)[: (dim // 2)].float() / dim) + ) + elif freqs_for == "pixel": + freqs = torch.linspace(1.0, max_freq / 2, dim // 2) * pi + elif freqs_for == "constant": + freqs = torch.ones(num_freqs).float() + else: + raise ValueError(f"unknown modality {freqs_for}") + + if ft_seq_len is None: + ft_seq_len = pt_seq_len + t = ( + torch.arange(ft_seq_len) / ft_seq_len * pt_seq_len + 1 + ) # + 1 is hacking vev0 pt code + + freqs = torch.einsum("..., f -> ... f", t, freqs) + freqs = repeat(freqs, "... n -> ... (n r)", r=2) + # freqs = broadcat((freqs[:, None, :], freqs[None, :, :]), dim = -1) + freqs = broadcat( + (freqs[None, :, :], freqs[:, None, :]), dim=-1 + ) # follow vev0 pt code + + freqs_cos = freqs.cos().view(-1, freqs.shape[-1]) + freqs_sin = freqs.sin().view(-1, freqs.shape[-1]) + + self.register_buffer("freqs_cos", freqs_cos) + self.register_buffer("freqs_sin", freqs_sin) + + print("======== shape of rope freq", self.freqs_cos.shape, "========") + + def forward(self, t): + return t * self.freqs_cos + rotate_half(t) * self.freqs_sin + + +class LayerNorm(nn.LayerNorm): + """Subclass torch's LayerNorm to handle fp16.""" + + def forward(self, x: torch.Tensor): + orig_type = x.dtype + # ret = super().forward(x.type(torch.float32)) + ret = F.layer_norm( + x.type(torch.float32), + self.normalized_shape, + self.weight.type(torch.float32), + self.bias.type(torch.float32), + self.eps, + ) + return ret.type(orig_type) + + +class QuickGELU(nn.Module): + def forward(self, x: torch.Tensor): + return x * torch.sigmoid(1.702 * x) + + +def drop_path( + x, drop_prob: float = 0.0, training: bool = False, scale_by_keep: bool = True +): + """Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). + + This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, + the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... + See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for + changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use + 'survival rate' as the argument. + + """ + if drop_prob == 0.0 or not training: + return x + keep_prob = 1 - drop_prob + shape = (x.shape[0],) + (1,) * ( + x.ndim - 1 + ) # work with diff dim tensors, not just 2D ConvNets + random_tensor = x.new_empty(shape).bernoulli_(keep_prob) + if keep_prob > 0.0 and scale_by_keep: + random_tensor.div_(keep_prob) + return x * random_tensor + + +class DropPath(nn.Module): + """Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).""" + + def __init__(self, drop_prob: float = 0.0, scale_by_keep: bool = True): + super(DropPath, self).__init__() + self.drop_prob = drop_prob + self.scale_by_keep = scale_by_keep + + def forward(self, x): + return drop_path(x, self.drop_prob, self.training, self.scale_by_keep) + + def extra_repr(self): + return f"drop_prob={round(self.drop_prob,3):0.3f}" + + +class Attention(nn.Module): + r""" + Implements attention based on Rope + """ + + def __init__( + self, + embed_dim: int, + num_heads: int, + dropout: float = 0.0, + bias: bool = True, + add_bias_kv: bool = False, + kdim: Optional[bool] = None, + vdim: Optional[bool] = None, + rope=None, + ): + super(Attention, self).__init__() + self.embed_dim = embed_dim + self.kdim = kdim if kdim is not None else embed_dim + self.vdim = vdim if vdim is not None else embed_dim + self._qkv_same_embed_dim = self.kdim == embed_dim and self.vdim == embed_dim + + self.num_heads = num_heads + self.dropout = dropout + self.head_dim = embed_dim // num_heads + assert ( + self.head_dim * num_heads == self.embed_dim + ), "embed_dim must be divisible by num_heads" + + if self._qkv_same_embed_dim is False: + self.q_proj_weight = Parameter(torch.Tensor(embed_dim, embed_dim)) + self.k_proj_weight = Parameter(torch.Tensor(embed_dim, self.kdim)) + self.v_proj_weight = Parameter(torch.Tensor(embed_dim, self.vdim)) + else: + self.in_proj_weight = Parameter(torch.empty(3 * embed_dim, embed_dim)) + + if bias: + self.in_proj_bias = Parameter(torch.empty(3 * embed_dim)) + else: + self.register_parameter("in_proj_bias", None) + self.out_proj = nn.Linear(embed_dim, embed_dim, bias=bias) + + if add_bias_kv: + self.bias_k = Parameter(torch.empty(1, 1, embed_dim)) + self.bias_v = Parameter(torch.empty(1, 1, embed_dim)) + else: + self.bias_k = self.bias_v = None + + self.rope = rope + + self.scale = self.head_dim ** (-0.5) + + version = torch.__version__ + # Split the version string and convert to a tuple of integers + version_tuple = tuple(map(int, version.split(".")[:2])) + # Check if the version is above 2.0 + if version_tuple >= (2, 0): + self.flash_att = True + else: + self.flash_att = False + + def forward(self, query, attn_mask: Optional[torch.Tensor] = None): + batch, seq, embed_dim = query.shape + + proj = torch._C._nn.linear(query, self.in_proj_weight, self.in_proj_bias) + # reshape to 3, E and not E, 3 is deliberate for better memory coalescing and keeping same order as chunk() + proj = ( + proj.unflatten(-1, (3, embed_dim)) + .unsqueeze(0) + .transpose(0, -2) + .squeeze(-2) + .contiguous() + ) + q_, k_, v_ = proj[0], proj[1], proj[2] + + # Use "q_" so that we don't accidentally quit in pdb :) + q_ = rearrange(q_, "b s (h d) -> b h s d", h=self.num_heads) + k_ = rearrange(k_, "b s (h d) -> b h s d", h=self.num_heads) + v_ = rearrange(v_, "b s (h d) -> b h s d", h=self.num_heads) + + ## rope + q_ = self.rope(q_).type_as(v_) + k_ = self.rope(k_).type_as(v_) + + if self.flash_att: + x_ = torch._C._nn.scaled_dot_product_attention( + q_, + k_, + v_, + attn_mask=attn_mask, + dropout_p=0.0, + is_causal=False, + scale=self.scale, + ) + else: + attn = (q_ * self.scale) @ k_.transpose(-2, -1) + attn = attn.softmax(dim=-1) + x_ = attn @ v_ + + x_ = rearrange(x_, "b h s d -> b s (h d)") + + return torch._C._nn.linear(x_, self.out_proj.weight, self.out_proj.bias) + + +class LayerScale(nn.Module): + def __init__( + self, + dim: int, + init_values: float = 1e-5, + inplace: bool = False, + ) -> None: + super().__init__() + self.inplace = inplace + self.gamma = nn.Parameter(init_values * torch.ones(dim)) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + return x.mul_(self.gamma) if self.inplace else x * self.gamma + + +class ResidualAttentionBlock(nn.Module): + def __init__( + self, + d_model: int, + n_head: int, + mlp_ratio=4.0, + act_layer=nn.GELU, + norm_layer=LayerNorm, + init_values=None, + drop_path=0.0, + use_rel_pos=False, + rel_pos_zero_init=True, + window_size=0, + rope=None, + input_size=None, + attn_mask=None, + ): + super().__init__() + + self.attn = Attention(embed_dim=d_model, num_heads=n_head, rope=rope) + self.ls_1 = ( + LayerScale(d_model, init_values=init_values) + if init_values + else nn.Identity() + ) + self.ln_1 = LayerNorm(d_model) + self.mlp = nn.Sequential( + OrderedDict( + [ + ("c_fc", nn.Linear(d_model, int(d_model * mlp_ratio))), + ("gelu", act_layer()), + ("c_proj", nn.Linear(int(d_model * mlp_ratio), d_model)), + ] + ) + ) + + self.drop_path = DropPath(drop_path) if drop_path > 0.0 else nn.Identity() + self.ln_2 = LayerNorm(d_model) + self.attn_mask = attn_mask + self.ls_2 = ( + LayerScale(d_model, init_values=init_values) + if init_values + else nn.Identity() + ) + self.window_size = window_size + + def attention_nhwc(self, x: torch.Tensor): + self.attn_mask = ( + self.attn_mask.to(dtype=x.dtype, device=x.device) + if self.attn_mask is not None + else None + ) + B, H, W, _ = x.shape + x = x.reshape(B, H * W, -1) + x = self.attn(x, attn_mask=self.attn_mask) + x = x.reshape(B, H, W, -1) + return x + + def forward(self, x: torch.Tensor): + shortcut = x + + x = self.ln_1(x) + # Window partition + if self.window_size > 0: + H, W = x.shape[1], x.shape[2] + x, pad_hw = window_partition(x, self.window_size) + + x = self.attention_nhwc(x) + # Reverse window partition + if self.window_size > 0: + x = window_unpartition(x, self.window_size, pad_hw, (H, W)) + + x = shortcut + self.drop_path(self.ls_1(x)) + x = x + self.drop_path(self.ls_2(self.mlp(self.ln_2(x)))) + return x + + +class Transformer(nn.Module): + def __init__( + self, + embed_dim: int, + depth: int, + num_heads: int, + mlp_ratio=4.0, + act_layer=nn.GELU, + norm_layer=LayerNorm, + init_values=None, + drop_path_rate=0.0, + use_rel_pos=False, + rel_pos_zero_init=True, + window_size=0, + window_block_indexes=(), + img_size=1024, + patch_size=16, + rope_win=None, + rope_glb=None, + use_act_checkpoint=False, + attn_mask=None, + ): + super().__init__() + self.use_act_checkpoint = use_act_checkpoint + + # stochastic depth decay rule + dpr = [x.item() for x in torch.linspace(0, drop_path_rate, depth)] + + self.resblocks = nn.ModuleList() + for i in range(depth): + block = ResidualAttentionBlock( + embed_dim, + num_heads, + attn_mask=attn_mask, + drop_path=dpr[i], + mlp_ratio=mlp_ratio, + act_layer=act_layer, + norm_layer=norm_layer, + init_values=init_values, + use_rel_pos=use_rel_pos, + rel_pos_zero_init=rel_pos_zero_init, + window_size=window_size if i in window_block_indexes else 0, + rope=rope_win if i in window_block_indexes else rope_glb, + input_size=(img_size // patch_size, img_size // patch_size), + ) + self.resblocks.append(block) + + def forward(self, x: torch.Tensor): + for idx, blk in enumerate(self.resblocks): + if self.use_act_checkpoint: + x = checkpoint(blk, x) + else: + x = blk(x) + return x + + +class PEv1_det(Backbone): + def __init__( + self, + img_size=1024, + patch_size=16, + in_chans=3, + embed_dim=768, + depth=12, + num_heads=12, + mlp_ratio=4.0, + qkv_bias=True, + drop_path_rate=0.0, + norm_layer=nn.LayerNorm, + act_layer=nn.GELU, + init_values=None, + use_abs_pos=True, + use_rel_pos=False, + rel_pos_zero_init=True, + rope=True, + pt_hw_seq_len=16, + intp_freq=True, + window_size=0, + window_block_indexes=(), + residual_block_indexes=(), + use_act_checkpoint=False, + pretrain_img_size=336, + pretrain_use_cls_token=True, + out_feature="last_feat", + tile_posemb=False, + ): + super().__init__() + self.pretrain_use_cls_token = pretrain_use_cls_token + + self.conv1 = nn.Conv2d( + in_channels=in_chans, + out_channels=embed_dim, + kernel_size=patch_size, + stride=patch_size, + bias=False, + ) + + if use_abs_pos: + # Initialize absolute positional embedding with pretrain image size. + num_patches = (pretrain_img_size // patch_size) * ( + pretrain_img_size // patch_size + ) + num_positions = (num_patches + 1) if pretrain_use_cls_token else num_patches + self.positional_embedding = nn.Parameter( + torch.zeros(1, num_positions, embed_dim) + ) + else: + self.positional_embedding = None + + self.tile_posemb = tile_posemb + + self.ln_pre = LayerNorm(embed_dim) + + half_head_dim = embed_dim // num_heads // 2 + hw_seq_len = img_size // patch_size + + self.rope_win = VisionRotaryEmbeddingFast( + dim=half_head_dim, + pt_seq_len=pt_hw_seq_len, + ft_seq_len=window_size if intp_freq else None, + ) + + self.rope_glb = VisionRotaryEmbeddingFast( + dim=half_head_dim, + pt_seq_len=pt_hw_seq_len, + ft_seq_len=hw_seq_len if intp_freq else None, + ) + + self.transformer = Transformer( + embed_dim=embed_dim, + depth=depth, + num_heads=num_heads, + mlp_ratio=mlp_ratio, + act_layer=act_layer, + norm_layer=norm_layer, + init_values=init_values, + drop_path_rate=drop_path_rate, + use_rel_pos=use_rel_pos, + rel_pos_zero_init=rel_pos_zero_init, + window_size=window_size, + window_block_indexes=window_block_indexes, + rope_win=self.rope_win, + rope_glb=self.rope_glb, + img_size=img_size, + patch_size=patch_size, + use_act_checkpoint=use_act_checkpoint, + ) + + self._out_feature_channels = {out_feature: embed_dim} + self._out_feature_strides = {out_feature: patch_size} + self._out_features = [out_feature] + + if self.positional_embedding is not None: + nn.init.trunc_normal_(self.positional_embedding, std=0.02) + + self.apply(self._init_weights) + + def _init_weights(self, m): + if isinstance(m, nn.Linear): + nn.init.trunc_normal_(m.weight, std=0.02) + if isinstance(m, nn.Linear) and m.bias is not None: + nn.init.constant_(m.bias, 0) + elif isinstance(m, nn.LayerNorm): + nn.init.constant_(m.bias, 0) + nn.init.constant_(m.weight, 1.0) + + def forward(self, x): + x = self.conv1(x) + x = x.permute(0, 2, 3, 1) + + if self.positional_embedding is not None: + x = x + get_abs_pos( + self.positional_embedding, + self.pretrain_use_cls_token, + (x.shape[1], x.shape[2]), + self.tile_posemb, + ) + x = self.ln_pre(x) + + x = self.transformer(x) + + outputs = {self._out_features[0]: x.permute(0, 3, 1, 2)} + return outputs + + +def get_vit_lr_decay_rate_pev1(name, lr_decay_rate=1.0, num_layers=12): + """ + Calculate lr decay rate for different ViT blocks. + Args: + name (string): parameter name. + lr_decay_rate (float): base lr decay rate. + num_layers (int): number of ViT blocks. + + Returns: + lr decay rate for the given parameter. + """ + layer_id = num_layers + 1 + if name.startswith("backbone"): + if ".positional_embedding" in name or ".conv1" in name or ".ln_pre" in name: + layer_id = 0 + elif ".resblocks." in name: + layer_id = int(name[name.find(".resblocks.") :].split(".")[2]) + 1 + return lr_decay_rate ** (num_layers + 1 - layer_id) diff --git a/perception_models/apps/detection/projects/ViTDet/configs/COCO/mask_rcnn_PEcore_G_coco75ep.py b/perception_models/apps/detection/projects/ViTDet/configs/COCO/mask_rcnn_PEcore_G_coco75ep.py new file mode 100644 index 0000000000000000000000000000000000000000..979f9b1fc66b1c739b809650f6b18d8c2c248c26 --- /dev/null +++ b/perception_models/apps/detection/projects/ViTDet/configs/COCO/mask_rcnn_PEcore_G_coco75ep.py @@ -0,0 +1,94 @@ +from functools import partial + +import torch.nn as nn +from detectron2 import model_zoo +from detectron2.config import LazyCall as L +from detectron2.modeling import SimpleFeaturePyramid, ViT +from detectron2.modeling.backbone.fpn import LastLevelMaxPool +from detectron2.solver import WarmupParamScheduler +from detectron2_pe.modeling import PEv1_det, get_vit_lr_decay_rate_pev1 +from fvcore.common.param_scheduler import MultiStepParamScheduler + +from ..COCO.mask_rcnn_vitdet_b_100ep import ( # dataloader,; model,; get_vit_lr_decay_rate, + lr_multiplier, optimizer, train) +from ..common.coco_loader_lsj import dataloader + +train.init_checkpoint = "/checkpoint/vision_encoder/pev1/pev1_rc2_d2.pt" +train.output_dir = ( + "/checkpoint/vision_encoder/d2_output/coco/mask_rcnn_PEcore_G_coco36ep" +) + +model = model_zoo.get_config("common/models/mask_rcnn_vitdet.py").model + +model.pixel_mean = [127, 127, 127] +model.pixel_std = [127, 127, 127] +model.input_format = "RGB" + + +img_size = 1024 +embed_dim, depth, num_heads, mlp_ratio, dp = 1536, 50, 16, 8960 / 1536, 0.5 +pretrain_img_size, patch_size, window_size = 512, 16, 32 +# 12, 24, 36, 49 for global attention +window_block_indexes = ( + list(range(0, 12)) + list(range(13, 24)) + list(range(25, 36)) + list(range(37, 49)) +) +# Creates Simple Feature Pyramid from ViT backbone +model.backbone = L(SimpleFeaturePyramid)( + net=L(PEv1_det)( # Single-scale ViT backbone + pretrain_img_size=pretrain_img_size, + img_size=img_size, + patch_size=patch_size, + embed_dim=embed_dim, + depth=depth, + num_heads=num_heads, + drop_path_rate=dp, + window_size=window_size, + pt_hw_seq_len=32, + mlp_ratio=mlp_ratio, + qkv_bias=True, + norm_layer=partial(nn.LayerNorm, eps=1e-6), + window_block_indexes=window_block_indexes, + residual_block_indexes=[], + use_rel_pos=True, + out_feature="last_feat", + tile_posemb=True, + use_abs_pos=True, + pretrain_use_cls_token=False, + use_act_checkpoint=True, + ), + in_feature="${.net.out_feature}", + out_channels=256, + scale_factors=(4.0, 2.0, 1.0, 0.5), + top_block=L(LastLevelMaxPool)(), + norm="LN", + square_pad=img_size, +) + +optimizer.params.lr_factor_func = partial( + get_vit_lr_decay_rate_pev1, lr_decay_rate=0.9, num_layers=50 +) + +dataloader.train.total_batch_size = 64 +# 100 ep = 184375 iters * 64 images/iter / 118000 images/ep +train.max_iter = 184375 + + +lr_multiplier = L(WarmupParamScheduler)( + scheduler=L(MultiStepParamScheduler)( + values=[1.0, 0.1, 0.01], + milestones=[163889, 177546], + num_updates=train.max_iter, + ), + warmup_length=250 / train.max_iter, + warmup_factor=0.001, +) + +optimizer.params.overrides = {} +optimizer.params.weight_decay_norm = None +optimizer.lr = 5e-5 + +train.max_iter = train.max_iter * 3 // 4 # 100ep -> 75ep +lr_multiplier.scheduler.milestones = [ + milestone * 3 // 4 for milestone in lr_multiplier.scheduler.milestones +] +lr_multiplier.scheduler.num_updates = train.max_iter diff --git a/perception_models/apps/detection/projects/ViTDet/configs/COCO/mask_rcnn_PEspatial_G_coco36ep.py b/perception_models/apps/detection/projects/ViTDet/configs/COCO/mask_rcnn_PEspatial_G_coco36ep.py new file mode 100644 index 0000000000000000000000000000000000000000..636beb25542a70bd87dc4c1e9a529223ce4782bf --- /dev/null +++ b/perception_models/apps/detection/projects/ViTDet/configs/COCO/mask_rcnn_PEspatial_G_coco36ep.py @@ -0,0 +1,95 @@ +from functools import partial + +import torch.nn as nn +from detectron2 import model_zoo +from detectron2.config import LazyCall as L +from detectron2.modeling import SimpleFeaturePyramid, ViT +from detectron2.modeling.backbone.fpn import LastLevelMaxPool +from detectron2.solver import WarmupParamScheduler +from detectron2_pe.modeling import PEv1_det, get_vit_lr_decay_rate_pev1 +from fvcore.common.param_scheduler import MultiStepParamScheduler + +from ..COCO.mask_rcnn_vitdet_b_100ep import ( # dataloader,; model,; get_vit_lr_decay_rate, + lr_multiplier, optimizer, train) +from ..common.coco_loader_lsj import dataloader + +train.init_checkpoint = "/checkpoint/vision_encoder/pev1/pev1_rc2_spatial_d2.pt" +train.output_dir = ( + "/checkpoint/vision_encoder/d2_output/coco/mask_rcnn_PEspatial_G_coco36ep" +) + +model = model_zoo.get_config("common/models/mask_rcnn_vitdet.py").model + +model.pixel_mean = [127, 127, 127] +model.pixel_std = [127, 127, 127] +model.input_format = "RGB" + + +img_size = 1024 +embed_dim, depth, num_heads, mlp_ratio, dp = 1536, 50, 16, 8960 / 1536, 0.5 +pretrain_img_size, patch_size, window_size = 512, 16, 32 +# 12, 24, 36, 49 for global attention +window_block_indexes = ( + list(range(0, 12)) + list(range(13, 24)) + list(range(25, 36)) + list(range(37, 49)) +) +# Creates Simple Feature Pyramid from ViT backbone +model.backbone = L(SimpleFeaturePyramid)( + net=L(PEv1_det)( # Single-scale ViT backbone + pretrain_img_size=pretrain_img_size, + img_size=img_size, + patch_size=patch_size, + embed_dim=embed_dim, + depth=depth, + num_heads=num_heads, + drop_path_rate=dp, + window_size=window_size, + pt_hw_seq_len=32, + mlp_ratio=mlp_ratio, + qkv_bias=True, + norm_layer=partial(nn.LayerNorm, eps=1e-6), + window_block_indexes=window_block_indexes, + residual_block_indexes=[], + use_rel_pos=True, + out_feature="last_feat", + tile_posemb=True, + use_abs_pos=True, + pretrain_use_cls_token=False, + use_act_checkpoint=True, + init_values=0.1, + ), + in_feature="${.net.out_feature}", + out_channels=256, + scale_factors=(4.0, 2.0, 1.0, 0.5), + top_block=L(LastLevelMaxPool)(), + norm="LN", + square_pad=img_size, +) + +optimizer.params.lr_factor_func = partial( + get_vit_lr_decay_rate_pev1, lr_decay_rate=0.9, num_layers=50 +) + +dataloader.train.total_batch_size = 64 +# 100 ep = 184375 iters * 64 images/iter / 118000 images/ep +train.max_iter = 184375 + + +lr_multiplier = L(WarmupParamScheduler)( + scheduler=L(MultiStepParamScheduler)( + values=[1.0, 0.1, 0.01], + milestones=[163889, 177546], + num_updates=train.max_iter, + ), + warmup_length=250 / train.max_iter, + warmup_factor=0.001, +) + +optimizer.params.overrides = {} +optimizer.params.weight_decay_norm = None +optimizer.lr = 5e-5 + +train.max_iter = train.max_iter * 36 // 100 # 100ep -> 36ep +lr_multiplier.scheduler.milestones = [ + milestone * 36 // 100 for milestone in lr_multiplier.scheduler.milestones +] +lr_multiplier.scheduler.num_updates = train.max_iter diff --git a/perception_models/apps/detection/projects/ViTDet/configs/COCO/mask_rcnn_vitdet_b_100ep.py b/perception_models/apps/detection/projects/ViTDet/configs/COCO/mask_rcnn_vitdet_b_100ep.py new file mode 100644 index 0000000000000000000000000000000000000000..750dd527521e3ae2edee316681cece7fa9f5d12b --- /dev/null +++ b/perception_models/apps/detection/projects/ViTDet/configs/COCO/mask_rcnn_vitdet_b_100ep.py @@ -0,0 +1,39 @@ +from functools import partial + +from detectron2 import model_zoo +from detectron2.config import LazyCall as L +from detectron2.modeling.backbone.vit import get_vit_lr_decay_rate +from detectron2.solver import WarmupParamScheduler +from fvcore.common.param_scheduler import MultiStepParamScheduler + +from ..common.coco_loader_lsj import dataloader + +model = model_zoo.get_config("common/models/mask_rcnn_vitdet.py").model + +# Initialization and trainer settings +train = model_zoo.get_config("common/train.py").train +train.amp.enabled = True +train.ddp.fp16_compression = True +train.init_checkpoint = "detectron2://ImageNetPretrained/MAE/mae_pretrain_vit_base.pth?matching_heuristics=True" + + +# Schedule +# 100 ep = 184375 iters * 64 images/iter / 118000 images/ep +train.max_iter = 184375 + +lr_multiplier = L(WarmupParamScheduler)( + scheduler=L(MultiStepParamScheduler)( + values=[1.0, 0.1, 0.01], + milestones=[163889, 177546], + num_updates=train.max_iter, + ), + warmup_length=250 / train.max_iter, + warmup_factor=0.001, +) + +# Optimizer +optimizer = model_zoo.get_config("common/optim.py").AdamW +optimizer.params.lr_factor_func = partial( + get_vit_lr_decay_rate, num_layers=12, lr_decay_rate=0.7 +) +optimizer.params.overrides = {"pos_embed": {"weight_decay": 0.0}} diff --git a/perception_models/apps/detection/projects/ViTDet/configs/LVIS/mask_rcnn_PEcore_G_lvis75ep.py b/perception_models/apps/detection/projects/ViTDet/configs/LVIS/mask_rcnn_PEcore_G_lvis75ep.py new file mode 100644 index 0000000000000000000000000000000000000000..27350aa4c14b97da48f6521a29efcfd64022658b --- /dev/null +++ b/perception_models/apps/detection/projects/ViTDet/configs/LVIS/mask_rcnn_PEcore_G_lvis75ep.py @@ -0,0 +1,121 @@ +from functools import partial + +import torch.nn as nn +from detectron2 import model_zoo +from detectron2.config import LazyCall as L +from detectron2.data.detection_utils import get_fed_loss_cls_weights +from detectron2.data.samplers import RepeatFactorTrainingSampler +from detectron2.evaluation.lvis_evaluation import LVISEvaluator +from detectron2.modeling import SimpleFeaturePyramid, ViT +from detectron2.modeling.backbone.fpn import LastLevelMaxPool +from detectron2.solver import WarmupParamScheduler +from detectron2_pe.modeling import PEv1_det, get_vit_lr_decay_rate_pev1 +from fvcore.common.param_scheduler import MultiStepParamScheduler + +from ..COCO.mask_rcnn_vitdet_b_100ep import ( # dataloader,; model,; get_vit_lr_decay_rate, + lr_multiplier, optimizer, train) +from ..common.coco_loader_lsj import dataloader + +train.init_checkpoint = "/checkpoint/vision_encoder/pev1/pev1_rc2_d2.pt" +train.output_dir = ( + "/checkpoint/vision_encoder/d2_output/lvis/mask_rcnn_PEcore_G_lvis75ep" +) + +model = model_zoo.get_config("common/models/mask_rcnn_vitdet.py").model + +model.pixel_mean = [127, 127, 127] +model.pixel_std = [127, 127, 127] +model.input_format = "RGB" + +img_size = 1024 +embed_dim, depth, num_heads, mlp_ratio, dp = 1536, 50, 16, 8960 / 1536, 0.5 +pretrain_img_size, patch_size, window_size = 512, 16, 32 +# 12, 24, 36, 49 for global attention +window_block_indexes = ( + list(range(0, 12)) + list(range(13, 24)) + list(range(25, 36)) + list(range(37, 49)) +) +# Creates Simple Feature Pyramid from ViT backbone +model.backbone = L(SimpleFeaturePyramid)( + net=L(PEv1_det)( # Single-scale ViT backbone + pretrain_img_size=pretrain_img_size, + img_size=img_size, + patch_size=patch_size, + embed_dim=embed_dim, + depth=depth, + num_heads=num_heads, + drop_path_rate=dp, + window_size=window_size, + pt_hw_seq_len=32, + mlp_ratio=mlp_ratio, + qkv_bias=True, + norm_layer=partial(nn.LayerNorm, eps=1e-6), + window_block_indexes=window_block_indexes, + residual_block_indexes=[], + use_rel_pos=True, + out_feature="last_feat", + tile_posemb=True, + use_abs_pos=True, + pretrain_use_cls_token=False, + use_act_checkpoint=True, + ), + in_feature="${.net.out_feature}", + out_channels=256, + scale_factors=(4.0, 2.0, 1.0, 0.5), + top_block=L(LastLevelMaxPool)(), + norm="LN", + square_pad=img_size, +) + +model.roi_heads.num_classes = 1203 +model.roi_heads.box_predictor.test_score_thresh = 0.02 +model.roi_heads.box_predictor.test_topk_per_image = 300 +model.roi_heads.box_predictor.use_sigmoid_ce = True +model.roi_heads.box_predictor.use_fed_loss = True +model.roi_heads.box_predictor.get_fed_loss_cls_weights = ( + lambda: get_fed_loss_cls_weights(dataloader.train.dataset.names, 0.5) +) + +train.eval_period = 30000 + +optimizer.params.lr_factor_func = partial( + get_vit_lr_decay_rate_pev1, lr_decay_rate=0.9, num_layers=50 +) + + +dataloader.train.dataset.names = "lvis_v1_train" +dataloader.train.sampler = L(RepeatFactorTrainingSampler)( + repeat_factors=L( + RepeatFactorTrainingSampler.repeat_factors_from_category_frequency + )(dataset_dicts="${dataloader.train.dataset}", repeat_thresh=0.001) +) +dataloader.test.dataset.names = "lvis_v1_val" +dataloader.evaluator = L(LVISEvaluator)( + dataset_name="${..test.dataset.names}", + max_dets_per_image=300, + output_dir="${train.output_dir}", +) + +dataloader.train.total_batch_size = 64 + +train.max_iter = 184375 + + +lr_multiplier = L(WarmupParamScheduler)( + scheduler=L(MultiStepParamScheduler)( + values=[1.0, 0.1, 0.01], + milestones=[163889, 177546], + num_updates=train.max_iter, + ), + warmup_length=250 / train.max_iter, + warmup_factor=0.001, +) + +optimizer.params.overrides = {} +optimizer.params.weight_decay_norm = None +optimizer.lr = 5e-5 + +train.max_iter = train.max_iter * 3 // 4 # 100ep -> 75ep +lr_multiplier.scheduler.milestones = [ + milestone * 3 // 4 for milestone in lr_multiplier.scheduler.milestones +] +lr_multiplier.scheduler.num_updates = train.max_iter diff --git a/perception_models/apps/detection/projects/ViTDet/configs/LVIS/mask_rcnn_PEspatial_G_lvis75ep.py b/perception_models/apps/detection/projects/ViTDet/configs/LVIS/mask_rcnn_PEspatial_G_lvis75ep.py new file mode 100644 index 0000000000000000000000000000000000000000..4ea75f68e505310c3c5a90baf0b0070df1a8400b --- /dev/null +++ b/perception_models/apps/detection/projects/ViTDet/configs/LVIS/mask_rcnn_PEspatial_G_lvis75ep.py @@ -0,0 +1,122 @@ +from functools import partial + +import torch.nn as nn +from detectron2 import model_zoo +from detectron2.config import LazyCall as L +from detectron2.data.detection_utils import get_fed_loss_cls_weights +from detectron2.data.samplers import RepeatFactorTrainingSampler +from detectron2.evaluation.lvis_evaluation import LVISEvaluator +from detectron2.modeling import SimpleFeaturePyramid, ViT +from detectron2.modeling.backbone.fpn import LastLevelMaxPool +from detectron2.solver import WarmupParamScheduler +from detectron2_pe.modeling import PEv1_det, get_vit_lr_decay_rate_pev1 +from fvcore.common.param_scheduler import MultiStepParamScheduler + +from ..COCO.mask_rcnn_vitdet_b_100ep import ( # dataloader,; model,; get_vit_lr_decay_rate, + lr_multiplier, optimizer, train) +from ..common.coco_loader_lsj import dataloader + +train.init_checkpoint = "/checkpoint/vision_encoder/pev1/pev1_rc2_spatial_d2.pt" +train.output_dir = ( + "/checkpoint/vision_encoder/d2_output/lvis/mask_rcnn_PEspatial_G_lvis75ep" +) + +model = model_zoo.get_config("common/models/mask_rcnn_vitdet.py").model + +model.pixel_mean = [127, 127, 127] +model.pixel_std = [127, 127, 127] +model.input_format = "RGB" + +img_size = 1024 +embed_dim, depth, num_heads, mlp_ratio, dp = 1536, 50, 16, 8960 / 1536, 0.5 +pretrain_img_size, patch_size, window_size = 512, 16, 32 +# 12, 24, 36, 49 for global attention +window_block_indexes = ( + list(range(0, 12)) + list(range(13, 24)) + list(range(25, 36)) + list(range(37, 49)) +) +# Creates Simple Feature Pyramid from ViT backbone +model.backbone = L(SimpleFeaturePyramid)( + net=L(PEv1_det)( # Single-scale ViT backbone + pretrain_img_size=pretrain_img_size, + img_size=img_size, + patch_size=patch_size, + embed_dim=embed_dim, + depth=depth, + num_heads=num_heads, + drop_path_rate=dp, + window_size=window_size, + pt_hw_seq_len=32, + mlp_ratio=mlp_ratio, + qkv_bias=True, + norm_layer=partial(nn.LayerNorm, eps=1e-6), + window_block_indexes=window_block_indexes, + residual_block_indexes=[], + use_rel_pos=True, + out_feature="last_feat", + tile_posemb=True, + use_abs_pos=True, + pretrain_use_cls_token=False, + use_act_checkpoint=True, + init_values=0.1, + ), + in_feature="${.net.out_feature}", + out_channels=256, + scale_factors=(4.0, 2.0, 1.0, 0.5), + top_block=L(LastLevelMaxPool)(), + norm="LN", + square_pad=img_size, +) + +model.roi_heads.num_classes = 1203 +model.roi_heads.box_predictor.test_score_thresh = 0.02 +model.roi_heads.box_predictor.test_topk_per_image = 300 +model.roi_heads.box_predictor.use_sigmoid_ce = True +model.roi_heads.box_predictor.use_fed_loss = True +model.roi_heads.box_predictor.get_fed_loss_cls_weights = ( + lambda: get_fed_loss_cls_weights(dataloader.train.dataset.names, 0.5) +) + +train.eval_period = 30000 + +optimizer.params.lr_factor_func = partial( + get_vit_lr_decay_rate_pev1, lr_decay_rate=0.9, num_layers=50 +) + + +dataloader.train.dataset.names = "lvis_v1_train" +dataloader.train.sampler = L(RepeatFactorTrainingSampler)( + repeat_factors=L( + RepeatFactorTrainingSampler.repeat_factors_from_category_frequency + )(dataset_dicts="${dataloader.train.dataset}", repeat_thresh=0.001) +) +dataloader.test.dataset.names = "lvis_v1_val" +dataloader.evaluator = L(LVISEvaluator)( + dataset_name="${..test.dataset.names}", + max_dets_per_image=300, + output_dir="${train.output_dir}", +) + +dataloader.train.total_batch_size = 64 + +train.max_iter = 184375 + + +lr_multiplier = L(WarmupParamScheduler)( + scheduler=L(MultiStepParamScheduler)( + values=[1.0, 0.1, 0.01], + milestones=[163889, 177546], + num_updates=train.max_iter, + ), + warmup_length=250 / train.max_iter, + warmup_factor=0.001, +) + +optimizer.params.overrides = {} +optimizer.params.weight_decay_norm = None +optimizer.lr = 5e-5 + +train.max_iter = train.max_iter * 3 // 4 # 100ep -> 75ep +lr_multiplier.scheduler.milestones = [ + milestone * 3 // 4 for milestone in lr_multiplier.scheduler.milestones +] +lr_multiplier.scheduler.num_updates = train.max_iter diff --git a/perception_models/apps/detection/projects/ViTDet/configs/common/coco_loader_lsj.py b/perception_models/apps/detection/projects/ViTDet/configs/common/coco_loader_lsj.py new file mode 100644 index 0000000000000000000000000000000000000000..e6c2f1e913a9f629290ce345fc4ffd4db4037e14 --- /dev/null +++ b/perception_models/apps/detection/projects/ViTDet/configs/common/coco_loader_lsj.py @@ -0,0 +1,22 @@ +import detectron2.data.transforms as T +from detectron2 import model_zoo +from detectron2.config import LazyCall as L + +# Data using LSJ +image_size = 1024 +dataloader = model_zoo.get_config("common/data/coco.py").dataloader +dataloader.train.mapper.augmentations = [ + L(T.RandomFlip)(horizontal=True), # flip first + L(T.ResizeScale)( + min_scale=0.1, max_scale=2.0, target_height=image_size, target_width=image_size + ), + L(T.FixedSizeCrop)(crop_size=(image_size, image_size), pad=False), +] +dataloader.train.mapper.image_format = "RGB" +dataloader.train.total_batch_size = 64 +# recompute boxes due to cropping +dataloader.train.mapper.recompute_boxes = True + +dataloader.test.mapper.augmentations = [ + L(T.ResizeShortestEdge)(short_edge_length=image_size, max_size=image_size), +] diff --git a/perception_models/apps/detection/scripts/coco/train_mask_rcnn_PEcore_G_coco75ep.sh b/perception_models/apps/detection/scripts/coco/train_mask_rcnn_PEcore_G_coco75ep.sh new file mode 100644 index 0000000000000000000000000000000000000000..b9215c4a5fa31205867510daa19e29d3356716e4 --- /dev/null +++ b/perception_models/apps/detection/scripts/coco/train_mask_rcnn_PEcore_G_coco75ep.sh @@ -0,0 +1,44 @@ +#!/bin/bash + +#SBATCH --qos=vision_encoder +#SBATCH --account=vision_encoder +#SBATCH --job-name=det +#SBATCH --nodes=8 +#SBATCH --ntasks=8 +#SBATCH --gres=gpu:8 +#SBATCH --cpus-per-task=32 +#SBATCH --mem=0 +#SBATCH --output=/checkpoint/vision_encoder/d2_output/slurm_logs/coco/train_mask_rcnn_PEcore_G_coco75ep/%j.out +#SBATCH --error=/checkpoint/vision_encoder/d2_output/slurm_logs/coco/train_mask_rcnn_PEcore_G_coco75ep/%j.err +#SBATCH --time=96:00:00 + +module load cuda/12.1 +nodes=($(scontrol show hostnames $SLURM_JOB_NODELIST)) +nodes_array=($nodes) +head_node=${nodes_array[0]} +head_node_ip=$(srun --nodes=1 --ntasks=1 -w "$head_node" hostname --ip-address) + +read -ra my_array <<< $head_node_ip +export LOGLEVEL=INFO + +echo head_node_ip $head_node_ip +echo endpoint "${head_node_ip}:29500" + +export DETECTRON2_DATASETS="/path/to/detectron2_data" +export PYTHONPATH="$HOME/occhi/apps/detection:$PYTHONPATH" + +srun \ +torchrun \ +--nnodes 8 \ +--nproc_per_node 8 \ +--rdzv_id $RANDOM \ +--rdzv_endpoint "${my_array[0]}:29500" \ +--rdzv_backend c10d \ +tools/lazyconfig_train_net_pe_slurm.py \ +--resume \ +--config-file projects/ViTDet/configs/COCO/mask_rcnn_PEcore_G_coco75ep.py \ +optimizer.lr=5e-5 \ +train.init_checkpoint="/checkpoint/vision_encoder/pev1/pe_core_G14_448_16patch.pt" \ +train.output_dir="/checkpoint/vision_encoder/d2_output/coco/train_mask_rcnn_PEcore_G_coco75ep" \ +model.backbone.net.use_act_checkpoint=True \ +"$@" diff --git a/perception_models/apps/detection/scripts/coco/train_mask_rcnn_PEspatial_G_coco36ep.sh b/perception_models/apps/detection/scripts/coco/train_mask_rcnn_PEspatial_G_coco36ep.sh new file mode 100644 index 0000000000000000000000000000000000000000..becdfae2693a0902755d8a1d8cafd2fff6e8d5bc --- /dev/null +++ b/perception_models/apps/detection/scripts/coco/train_mask_rcnn_PEspatial_G_coco36ep.sh @@ -0,0 +1,45 @@ +#!/bin/bash + +#SBATCH --qos=vision_encoder +#SBATCH --account=vision_encoder +#SBATCH --job-name=det +#SBATCH --nodes=8 +#SBATCH --ntasks=8 +#SBATCH --gres=gpu:8 +#SBATCH --cpus-per-task=32 +#SBATCH --mem=0 +#SBATCH --output=/checkpoint/vision_encoder/d2_output/slurm_logs/coco/train_mask_rcnn_PEspatial_G_coco36ep/%j.out +#SBATCH --error=/checkpoint/vision_encoder/d2_output/slurm_logs/coco/train_mask_rcnn_PEspatial_G_coco36ep/%j.err +#SBATCH --time=96:00:00 + +module load cuda/12.1 +nodes=($(scontrol show hostnames $SLURM_JOB_NODELIST)) +nodes_array=($nodes) +head_node=${nodes_array[0]} +head_node_ip=$(srun --nodes=1 --ntasks=1 -w "$head_node" hostname --ip-address) + +read -ra my_array <<< $head_node_ip +export LOGLEVEL=INFO + +echo head_node_ip $head_node_ip +echo endpoint "${head_node_ip}:29500" + +export DETECTRON2_DATASETS="/path/to/detectron2_data" +export PYTHONPATH="$HOME/occhi/apps/detection:$PYTHONPATH" + +srun \ +torchrun \ +--nnodes 8 \ +--nproc_per_node 8 \ +--rdzv_id $RANDOM \ +--rdzv_endpoint "${my_array[0]}:29500" \ +--rdzv_backend c10d \ +tools/lazyconfig_train_net_pe_slurm.py \ +--resume \ +--config-file projects/ViTDet/configs/COCO/mask_rcnn_PEspatial_G_coco36ep.py \ +optimizer.lr=5e-5 \ +train.init_checkpoint="/checkpoint/vision_encoder/pev1/pe_spatial_G14_16patch.pth" \ +train.output_dir="/checkpoint/vision_encoder/d2_output/coco/train_mask_rcnn_PEspatial_G_coco36ep" \ +model.backbone.net.init_values=0.1 \ +model.backbone.net.use_act_checkpoint=True \ +"$@" diff --git a/perception_models/apps/detection/scripts/evaluate_local.sh b/perception_models/apps/detection/scripts/evaluate_local.sh new file mode 100644 index 0000000000000000000000000000000000000000..b94214b61afbe586f8d3acd3e515803e7e2e0361 --- /dev/null +++ b/perception_models/apps/detection/scripts/evaluate_local.sh @@ -0,0 +1,7 @@ +export DETECTRON2_DATASETS="/path/to/detectron2_data" +export PYTHONPATH="$HOME/occhi/apps/detection:$PYTHONPATH" + +python3 tools/lazyconfig_train_net_pe.py \ +--num-gpus 8 \ +--eval-only \ +"$@" \ No newline at end of file diff --git a/perception_models/apps/detection/scripts/lvis/train_mask_rcnn_PEcore_G_lvis75ep.sh b/perception_models/apps/detection/scripts/lvis/train_mask_rcnn_PEcore_G_lvis75ep.sh new file mode 100644 index 0000000000000000000000000000000000000000..88d7551ac900a4899df88a5dd2b38354b01092ca --- /dev/null +++ b/perception_models/apps/detection/scripts/lvis/train_mask_rcnn_PEcore_G_lvis75ep.sh @@ -0,0 +1,44 @@ +#!/bin/bash + +#SBATCH --qos=vision_encoder +#SBATCH --account=vision_encoder +#SBATCH --job-name=det +#SBATCH --nodes=8 +#SBATCH --ntasks=8 +#SBATCH --gres=gpu:8 +#SBATCH --cpus-per-task=32 +#SBATCH --mem=0 +#SBATCH --output=/checkpoint/vision_encoder/d2_output/slurm_logs/lvis/train_mask_rcnn_PEcore_G_lvis75ep/%j.out +#SBATCH --error=/checkpoint/vision_encoder/d2_output/slurm_logs/lvis/train_mask_rcnn_PEcore_G_lvis75ep/%j.err +#SBATCH --time=96:00:00 + +module load cuda/12.1 +nodes=($(scontrol show hostnames $SLURM_JOB_NODELIST)) +nodes_array=($nodes) +head_node=${nodes_array[0]} +head_node_ip=$(srun --nodes=1 --ntasks=1 -w "$head_node" hostname --ip-address) + +read -ra my_array <<< $head_node_ip +export LOGLEVEL=INFO + +echo head_node_ip $head_node_ip +echo endpoint "${head_node_ip}:29500" + +export DETECTRON2_DATASETS="/path/to/detectron2_data" +export PYTHONPATH="$HOME/occhi/apps/detection:$PYTHONPATH" + +srun \ +torchrun \ +--nnodes 8 \ +--nproc_per_node 8 \ +--rdzv_id $RANDOM \ +--rdzv_endpoint "${my_array[0]}:29500" \ +--rdzv_backend c10d \ +tools/lazyconfig_train_net_pe_slurm.py \ +--resume \ +--config-file projects/ViTDet/configs/LVIS/mask_rcnn_PEcore_G_lvis75ep.py \ +optimizer.lr=5e-5 \ +train.init_checkpoint="/checkpoint/vision_encoder/pev1/pe_core_G14_448_16patch.pt" \ +train.output_dir="/checkpoint/vision_encoder/d2_output/lvis/train_mask_rcnn_PEcore_G_lvis75ep" \ +model.backbone.net.use_act_checkpoint=True \ +"$@" diff --git a/perception_models/apps/detection/scripts/lvis/train_mask_rcnn_PEspatial_G_lvis75ep.sh b/perception_models/apps/detection/scripts/lvis/train_mask_rcnn_PEspatial_G_lvis75ep.sh new file mode 100644 index 0000000000000000000000000000000000000000..7e9bccafbb20d744d88dd30442c32aaaa172a2c6 --- /dev/null +++ b/perception_models/apps/detection/scripts/lvis/train_mask_rcnn_PEspatial_G_lvis75ep.sh @@ -0,0 +1,45 @@ +#!/bin/bash + +#SBATCH --qos=vision_encoder +#SBATCH --account=vision_encoder +#SBATCH --job-name=det +#SBATCH --nodes=8 +#SBATCH --ntasks=8 +#SBATCH --gres=gpu:8 +#SBATCH --cpus-per-task=32 +#SBATCH --mem=0 +#SBATCH --output=/checkpoint/vision_encoder/d2_output/slurm_logs/lvis/train_mask_rcnn_PEspatial_G_lvis75ep/%j.out +#SBATCH --error=/checkpoint/vision_encoder/d2_output/slurm_logs/lvis/train_mask_rcnn_PEspatial_G_lvis75ep/%j.err +#SBATCH --time=96:00:00 + +module load cuda/12.1 +nodes=($(scontrol show hostnames $SLURM_JOB_NODELIST)) +nodes_array=($nodes) +head_node=${nodes_array[0]} +head_node_ip=$(srun --nodes=1 --ntasks=1 -w "$head_node" hostname --ip-address) + +read -ra my_array <<< $head_node_ip +export LOGLEVEL=INFO + +echo head_node_ip $head_node_ip +echo endpoint "${head_node_ip}:29500" + +export DETECTRON2_DATASETS="/path/to/detectron2_data" +export PYTHONPATH="$HOME/occhi/apps/detection:$PYTHONPATH" + +srun \ +torchrun \ +--nnodes 8 \ +--nproc_per_node 8 \ +--rdzv_id $RANDOM \ +--rdzv_endpoint "${my_array[0]}:29500" \ +--rdzv_backend c10d \ +tools/lazyconfig_train_net_pe_slurm.py \ +--resume \ +--config-file projects/ViTDet/configs/LVIS/mask_rcnn_PEspatial_G_lvis75ep.py \ +optimizer.lr=5e-5 \ +train.init_checkpoint="/checkpoint/vision_encoder/pev1/pe_spatial_G14_16patch.pth" \ +train.output_dir="/checkpoint/vision_encoder/d2_output/lvis/train_mask_rcnn_PEspatial_G_lvis75ep" \ +model.backbone.net.init_values=0.1 \ +model.backbone.net.use_act_checkpoint=True \ +"$@" diff --git a/perception_models/apps/detection/tools/convert_d2.py b/perception_models/apps/detection/tools/convert_d2.py new file mode 100644 index 0000000000000000000000000000000000000000..96ff850b575457f102e55d0515fe5eb629ce7d8e --- /dev/null +++ b/perception_models/apps/detection/tools/convert_d2.py @@ -0,0 +1,101 @@ +# Modified from https://github.com/baaivision/EVA/blob/master/EVA-01/eva/interpolate_patch_14to16.py +import argparse + +import torch + + +def interpolate_pos_embed( + checkpoint_model, key_name="pos_embed", new_patches=196, num_extra_tokens=1 +): + if key_name in checkpoint_model: + pos_embed_checkpoint = checkpoint_model[key_name] + if pos_embed_checkpoint.dim() == 2: + pos_embed_checkpoint = pos_embed_checkpoint.unsqueeze(0) + embedding_size = pos_embed_checkpoint.shape[-1] + # height (== width) for the checkpoint position embedding + orig_size = int((pos_embed_checkpoint.shape[-2] - num_extra_tokens) ** 0.5) + # height (== width) for the new position embedding + new_size = int(new_patches**0.5) + # class_token and dist_token are kept unchanged + if orig_size != new_size: + print( + "Position interpolate from %dx%d to %dx%d" + % (orig_size, orig_size, new_size, new_size) + ) + else: + print("Position interpolate is skipped as original size equals new size") + return + extra_tokens = pos_embed_checkpoint[:, :num_extra_tokens] + # only the position tokens are interpolated + pos_tokens = pos_embed_checkpoint[:, num_extra_tokens:] + pos_tokens = pos_tokens.reshape( + -1, orig_size, orig_size, embedding_size + ).permute(0, 3, 1, 2) + pos_tokens = torch.nn.functional.interpolate( + pos_tokens, size=(new_size, new_size), mode="bicubic", align_corners=False + ) + pos_tokens = pos_tokens.permute(0, 2, 3, 1).flatten(1, 2) + new_pos_embed = torch.cat((extra_tokens, pos_tokens), dim=1) + checkpoint_model[key_name] = new_pos_embed + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description="convert to d2 format") + parser.add_argument("--input", default="/path/to/input.pt", type=str) + parser.add_argument("--output", default="/path/to/input.pt", type=str) + parser.add_argument("--prefix", default="module.visual.", type=str) + parser.add_argument("--output_pixel", default=224, type=int) + parser.add_argument("--output_patch_size", default=16, type=int) + parser.add_argument("--num_extra_tokens", default=0, type=int) + parser.add_argument("--keep_pe", action="store_true") + args = parser.parse_args() + + checkpoint_ori = torch.load(args.input, map_location=torch.device("cpu"))[ + "state_dict" + ] + checkpoint = {} + + prefix = args.prefix + for k, v in checkpoint_ori.items(): + if k.startswith(prefix): + checkpoint[k[len(prefix) :]] = v + + # interpolate patch_embed + patch_embed = checkpoint["conv1.weight"] + C_o, C_in, H, W = patch_embed.shape + if H != args.output_patch_size or W != args.output_patch_size: + patch_embed = torch.nn.functional.interpolate( + patch_embed.float(), + size=(args.output_patch_size, args.output_patch_size), + mode="bicubic", + align_corners=False, + ) + checkpoint["conv1.weight"] = patch_embed + + # interpolate pos_embed too + if not args.keep_pe: + interpolate_pos_embed( + checkpoint, + key_name="positional_embedding", + new_patches=(args.output_pixel / args.output_patch_size) + * (args.output_pixel / args.output_patch_size), + num_extra_tokens=args.num_extra_tokens, + ) + else: + positional_embedding = checkpoint["positional_embedding"].unsqueeze(0) + checkpoint["positional_embedding"] = positional_embedding + + print("======== new state_dict ========") + for k, v in list(checkpoint.items()): + print(k, " ", v.shape) + + torch.save({"model": checkpoint}, args.output) + +""" +python3 tools/convert_d2.py --input /checkpoint/vision_encoder/pev1/pe_core_G14_448.pt --keep_pe --output /checkpoint/vision_encoder/pev1/pe_core_G14_448_16patch.pt +python3 tools/convert_d2.py --input /checkpoint/vision_encoder/pev1/pe_spatial_G14_448.pt --keep_pe --output /checkpoint/vision_encoder/pev1/pe_spatial_G14_16patch.pth + +python3 tools/convert_d2.py --input /checkpoint/vision_encoder/pev1/pe_spatial_G14_448.pt --output_pixel 224 --output /checkpoint/vision_encoder/pev1/pe_spatial_G14_448_16patch224pix.pth +python3 tools/convert_d2.py --input /checkpoint/vision_encoder/pev1/pe_spatial_G14_448.pt --output_pixel 384 --output /checkpoint/vision_encoder/pev1/pe_spatial_G14_448_16patch384pix.pth + +""" diff --git a/perception_models/apps/detection/tools/lazyconfig_train_net_pe.py b/perception_models/apps/detection/tools/lazyconfig_train_net_pe.py new file mode 100644 index 0000000000000000000000000000000000000000..fc11d408678d457033849bc0a942089d6a997efc --- /dev/null +++ b/perception_models/apps/detection/tools/lazyconfig_train_net_pe.py @@ -0,0 +1,138 @@ +#!/usr/bin/env python +# Copyright (c) Facebook, Inc. and its affiliates. +""" +Training script using the new "LazyConfig" python config files. + +This scripts reads a given python config file and runs the training or evaluation. +It can be used to train any models or dataset as long as they can be +instantiated by the recursive construction defined in the given config file. + +Besides lazy construction of models, dataloader, etc., this scripts expects a +few common configuration parameters currently defined in "configs/common/train.py". +To add more complicated training logic, you can easily add other configs +in the config file and implement a new train_net.py to handle them. +""" +import logging + +from detectron2.config import LazyConfig, instantiate +from detectron2.engine import (AMPTrainer, SimpleTrainer, + default_argument_parser, default_setup, + default_writers, hooks, launch) +from detectron2.engine.defaults import create_ddp_model +from detectron2.evaluation import inference_on_dataset, print_csv_format +from detectron2.utils import comm +# from detectron2.checkpoint import DetectionCheckpointer +from detectron2_pe.checkpoint import DetectionCheckpointer + +logger = logging.getLogger("detectron2") + + +def do_test(cfg, model): + if "evaluator" in cfg.dataloader: + ret = inference_on_dataset( + model, + instantiate(cfg.dataloader.test), + instantiate(cfg.dataloader.evaluator), + ) + print_csv_format(ret) + return ret + + +def do_train(args, cfg): + """ + Args: + cfg: an object with the following attributes: + model: instantiate to a module + dataloader.{train,test}: instantiate to dataloaders + dataloader.evaluator: instantiate to evaluator for test set + optimizer: instantaite to an optimizer + lr_multiplier: instantiate to a fvcore scheduler + train: other misc config defined in `configs/common/train.py`, including: + output_dir (str) + init_checkpoint (str) + amp.enabled (bool) + max_iter (int) + eval_period, log_period (int) + device (str) + checkpointer (dict) + ddp (dict) + """ + model = instantiate(cfg.model) + logger = logging.getLogger("detectron2") + logger.info("Model:\n{}".format(model)) + model.to(cfg.train.device) + + cfg.optimizer.params.model = model + optim = instantiate(cfg.optimizer) + + train_loader = instantiate(cfg.dataloader.train) + + model = create_ddp_model(model, **cfg.train.ddp) + trainer = (AMPTrainer if cfg.train.amp.enabled else SimpleTrainer)( + model, train_loader, optim + ) + checkpointer = DetectionCheckpointer( + model, + cfg.train.output_dir, + trainer=trainer, + ) + trainer.register_hooks( + [ + hooks.IterationTimer(), + hooks.LRScheduler(scheduler=instantiate(cfg.lr_multiplier)), + ( + hooks.PeriodicCheckpointer(checkpointer, **cfg.train.checkpointer) + if comm.is_main_process() + else None + ), + hooks.EvalHook(cfg.train.eval_period, lambda: do_test(cfg, model)), + ( + hooks.PeriodicWriter( + default_writers(cfg.train.output_dir, cfg.train.max_iter), + period=cfg.train.log_period, + ) + if comm.is_main_process() + else None + ), + ] + ) + + checkpointer.resume_or_load(cfg.train.init_checkpoint, resume=args.resume) + if args.resume and checkpointer.has_checkpoint(): + # The checkpoint stores the training iteration that just finished, thus we start + # at the next iteration + start_iter = trainer.iter + 1 + else: + start_iter = 0 + trainer.train(start_iter, cfg.train.max_iter) + + +def main(args): + cfg = LazyConfig.load(args.config_file) + cfg = LazyConfig.apply_overrides(cfg, args.opts) + default_setup(cfg, args) + + if args.eval_only: + model = instantiate(cfg.model) + model.to(cfg.train.device) + model = create_ddp_model(model) + DetectionCheckpointer(model).load(cfg.train.init_checkpoint) + print(do_test(cfg, model)) + else: + do_train(args, cfg) + + +def invoke_main() -> None: + args = default_argument_parser().parse_args() + launch( + main, + args.num_gpus, + num_machines=args.num_machines, + machine_rank=args.machine_rank, + dist_url=args.dist_url, + args=(args,), + ) + + +if __name__ == "__main__": + invoke_main() # pragma: no cover diff --git a/perception_models/apps/detection/tools/lazyconfig_train_net_pe_slurm.py b/perception_models/apps/detection/tools/lazyconfig_train_net_pe_slurm.py new file mode 100644 index 0000000000000000000000000000000000000000..c7219d1bb3292c68b689a26cc463c69c162b2371 --- /dev/null +++ b/perception_models/apps/detection/tools/lazyconfig_train_net_pe_slurm.py @@ -0,0 +1,242 @@ +#!/usr/bin/env python +# Copyright (c) Facebook, Inc. and its affiliates. +""" +Training script using the new "LazyConfig" python config files. + +This scripts reads a given python config file and runs the training or evaluation. +It can be used to train any models or dataset as long as they can be +instantiated by the recursive construction defined in the given config file. + +Besides lazy construction of models, dataloader, etc., this scripts expects a +few common configuration parameters currently defined in "configs/common/train.py". +To add more complicated training logic, you can easily add other configs +in the config file and implement a new train_net.py to handle them. +""" +import argparse +import logging +import os + +import torch +import torch.distributed as dist +from detectron2.config import LazyConfig, instantiate +from detectron2.engine import (AMPTrainer, SimpleTrainer, + default_argument_parser, default_setup, + default_writers, hooks, launch) +from detectron2.engine.defaults import create_ddp_model +from detectron2.evaluation import inference_on_dataset, print_csv_format +from detectron2.utils import comm +# from detectron2.checkpoint import DetectionCheckpointer +from detectron2_pe.checkpoint import DetectionCheckpointer + +logger = logging.getLogger("detectron2") + + +def do_test(cfg, model): + if "evaluator" in cfg.dataloader: + ret = inference_on_dataset( + model, + instantiate(cfg.dataloader.test), + instantiate(cfg.dataloader.evaluator), + ) + print_csv_format(ret) + return ret + + +def do_train(args, cfg): + """ + Args: + cfg: an object with the following attributes: + model: instantiate to a module + dataloader.{train,test}: instantiate to dataloaders + dataloader.evaluator: instantiate to evaluator for test set + optimizer: instantaite to an optimizer + lr_multiplier: instantiate to a fvcore scheduler + train: other misc config defined in `configs/common/train.py`, including: + output_dir (str) + init_checkpoint (str) + amp.enabled (bool) + max_iter (int) + eval_period, log_period (int) + device (str) + checkpointer (dict) + ddp (dict) + """ + model = instantiate(cfg.model) + logger = logging.getLogger("detectron2") + logger.info("Model:\n{}".format(model)) + model.to(cfg.train.device) + + cfg.optimizer.params.model = model + optim = instantiate(cfg.optimizer) + + train_loader = instantiate(cfg.dataloader.train) + + model = create_ddp_model(model, **cfg.train.ddp) + trainer = (AMPTrainer if cfg.train.amp.enabled else SimpleTrainer)( + model, train_loader, optim + ) + checkpointer = DetectionCheckpointer( + model, + cfg.train.output_dir, + trainer=trainer, + ) + trainer.register_hooks( + [ + hooks.IterationTimer(), + hooks.LRScheduler(scheduler=instantiate(cfg.lr_multiplier)), + ( + hooks.PeriodicCheckpointer(checkpointer, **cfg.train.checkpointer) + if comm.is_main_process() + else None + ), + hooks.EvalHook(cfg.train.eval_period, lambda: do_test(cfg, model)), + ( + hooks.PeriodicWriter( + default_writers(cfg.train.output_dir, cfg.train.max_iter), + period=cfg.train.log_period, + ) + if comm.is_main_process() + else None + ), + ] + ) + + checkpointer.resume_or_load(cfg.train.init_checkpoint, resume=args.resume) + if args.resume and checkpointer.has_checkpoint(): + # The checkpoint stores the training iteration that just finished, thus we start + # at the next iteration + start_iter = trainer.iter + 1 + else: + start_iter = 0 + trainer.train(start_iter, cfg.train.max_iter) + + +def main(args): + cfg = LazyConfig.load(args.config_file) + cfg = LazyConfig.apply_overrides(cfg, args.opts) + default_setup(cfg, args) + + if args.eval_only: + model = instantiate(cfg.model) + model.to(cfg.train.device) + model = create_ddp_model(model) + DetectionCheckpointer(model).load(cfg.train.init_checkpoint) + print(do_test(cfg, model)) + else: + do_train(args, cfg) + + +def invoke_main() -> None: + args = default_argument_parser().parse_args() + launch( + main, + args.num_gpus, + num_machines=args.num_machines, + machine_rank=args.machine_rank, + dist_url=args.dist_url, + args=(args,), + ) + + +def default_argument_parser_slurm(epilog=None): + """ + Create a parser with some common arguments used by detectron2 users. + + Args: + epilog (str): epilog passed to ArgumentParser describing the usage. + + Returns: + argparse.ArgumentParser: + """ + parser = argparse.ArgumentParser( + epilog=epilog or f"Launch using torchrun", + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + parser.add_argument( + "--config-file", default="", metavar="FILE", help="path to config file" + ) + parser.add_argument( + "--resume", + action="store_true", + help="Whether to attempt to resume from the checkpoint directory. " + "See documentation of `DefaultTrainer.resume_or_load()` for what it means.", + ) + parser.add_argument( + "--eval-only", action="store_true", help="perform evaluation only" + ) + # parser.add_argument("--num-gpus", type=int, default=1, help="number of gpus *per machine*") + # parser.add_argument("--num-machines", type=int, default=1, help="total number of machines") + # parser.add_argument( + # "--machine-rank", type=int, default=0, help="the rank of this machine (unique per machine)" + # ) + + # # PyTorch still may leave orphan processes in multi-gpu training. + # # Therefore we use a deterministic way to obtain port, + # # so that users are aware of orphan processes by seeing the port occupied. + # port = 2**15 + 2**14 + hash(os.getuid() if sys.platform != "win32" else 1) % 2**14 + # parser.add_argument( + # "--dist-url", + # default="tcp://127.0.0.1:{}".format(port), + # help="initialization URL for pytorch distributed backend. See " + # "https://pytorch.org/docs/stable/distributed.html for details.", + # ) + parser.add_argument( + "opts", + help=""" +Modify config options at the end of the command. For Yacs configs, use +space-separated "PATH.KEY VALUE" pairs. +For python-based LazyConfig, use "path.key=value". + """.strip(), + default=None, + nargs=argparse.REMAINDER, + ) + return parser + + +def main_worker(main_func, args): + world_size = int(os.environ["WORLD_SIZE"]) + rank = int(os.environ["RANK"]) + local_rank = int(os.environ["LOCAL_RANK"]) + + has_gpu = torch.cuda.is_available() + if has_gpu: + torch.cuda.set_device(local_rank) + dist.init_process_group( + backend="nccl" if has_gpu else "gloo", + init_method="env://", + ) + + # Setup the local process group + num_gpus_per_machine = torch.cuda.device_count() + machine_rank = rank // num_gpus_per_machine + + # comm.create_local_process_group(num_gpus_per_machine) + # Setup the local process group (which contains ranks within the same machine) + assert comm._LOCAL_PROCESS_GROUP is None + num_machines = world_size // num_gpus_per_machine + for i in range(num_machines): + ranks_on_i = list( + range(i * num_gpus_per_machine, (i + 1) * num_gpus_per_machine) + ) + pg = dist.new_group(ranks_on_i) + if i == machine_rank: + comm._LOCAL_PROCESS_GROUP = pg + + # synchronize to prevent possible timeout after calling init_process_group + comm.synchronize() + + main_func(args) + + +def slurm_main() -> None: + """ + Launch the main function using torchrun. + This function should be called in the main script. + """ + args = default_argument_parser_slurm().parse_args() + main_worker(main, args) + + +if __name__ == "__main__": + # invoke_main() # pragma: no cover + slurm_main() diff --git a/perception_models/apps/pe/README.md b/perception_models/apps/pe/README.md new file mode 100644 index 0000000000000000000000000000000000000000..349a2b58131b9c85272a79613fbbb44e1010c814 --- /dev/null +++ b/perception_models/apps/pe/README.md @@ -0,0 +1,319 @@ +# Perception Encoder (PE) + +[![Paper](https://img.shields.io/badge/Paper-Perception%20Encoder-b31b1b.svg)](https://ai.meta.com/research/publications/perception-encoder-the-best-visual-embeddings-are-not-at-the-output-of-the-network) +[![Paper](https://img.shields.io/badge/Technical%20Report-Perception%20Encoder-b31b1b.svg)](https://ai.meta.com/research/publications/pushing-the-frontier-of-audiovisual-perception-with-large-scale-multimodal-correspondence-learning/) +[![Paper](https://img.shields.io/badge/arXiv-2504.13181-brightgreen.svg?style=flat-square)](https://arxiv.org/abs/2504.13181) +[![Hugging Face Collection](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Collection-blue)](https://huggingface.co/collections/facebook/perception-encoder-67f977c9a65ca5895a7f6ba1) +[![Colab Demo](https://img.shields.io/static/v1?label=Demo&message=Google%20Colab&logo=google&color=orange)](https://colab.research.google.com/github/facebookresearch/perception_models/blob/main/apps/pe/docs/pe_demo.ipynb) +[![Model License](https://img.shields.io/badge/Model_License-Apache_2.0-olive)](https://opensource.org/licenses/Apache-2.0) + +This is the official implementation of **Perception Encoder** from our paper: +**[Perception Encoder: The best visual embeddings are hidden inside the network](https://ai.meta.com/research/publications/perception-encoder-the-best-visual-embeddings-are-not-at-the-output-of-the-network)** +Daniel Bolya*, Po-Yao Huang*, Peize Sun*, Jang Hyun Cho*, Andrea Madotto*, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, Junke Wang, Marco Monteiro, Hu Xu, Shiyu Dong, Nikhila Ravi, Daniel Li, Piotr Dollár, Christoph Feichtenhofer +\* Joint First Author +_[HuggingFace](https://huggingface.co/collections/facebook/perception-encoder-67f977c9a65ca5895a7f6ba1)_ | _[Blog](https://ai.meta.com/blog/meta-fair-updates-perception-localization-reasoning)_ | _[GitHub](https://github.com/facebookresearch/perception_models)_ | _[arXiv](https://arxiv.org/abs/2504.13181)_ | _[BibTeX](#citation)_ + + +
+ +Perception Encoder (PE) is a family of models that exhibits state-of-the-art performance on a large variety of vision tasks. By using a robust contrastive pretraining recipe and finetuning on synthetically aligned videos, PE not only outperforms all existing models on classification and retrieval, but it also internally produces strong, general features that scale for downstream tasks. PE unlocks the ability for large-scale contrastive pretraining to transfer to downstream tasks with alignment tuning to capitalize on those general features. + +The result is an extremely powerful family of checkpoints: PE core can outperform SigLIP2 on Image CLIP and InternVideo2 on Video CLIP; PE lang can be used to outperform QwenVL2.5 and InternVL3 on multimodal language modeling; and PE spatial can outperform DINOv2 on dense prediction tasks—all following the same, easily scalable contrastive pretraining. + + +### Contents +PE has 4 types of checkpoints, each excelling in a different area of computer vision: + - [PE core](#perception-encoder-core): a state-of-the-art CLIP model for zero-shot image and video classification as well as image and video retrieval. + - [PE lang](#perception-encoder-language): a state-of-the-art large language model aligned vision encoder that enables our open-data [Perception Language Model (PLM)](../plm/README.md) to compete at the forefront of the field. + - [PE spatial](#perception-encoder-spatial): a state-of-the-art spatially tuned model that can outperform the best spatial models for dense prediction tasks like detection, depth estimation, and tracking. + - [PE-AV](#perception-encoder-audiovisual): a state-of-the-art spatially tuned model that can outperform the best spatial models for dense prediction tasks like detection, depth estimation, and tracking. + +Finally, we also release a dataset we collected in the process of creating our novel video data engine: + - [PE Video Dataset (PVD)](#pe-video-dataset-pvd): an diverse set of 1M high quality datasets with accompanying metadata as well as 120K human-refined detailed video captions. + +If you want to get started right away check out the [usage](#usage) section! + +:construction: This repository is under construction! :construction: + + +## Perception Encoder: Core +PE core is our base model trained with our robust image pretraining schedule and finetuned on the data generated by our synthetic video data engine. + +#### Model Configurations +PE core curently comes in 3 sizes. PE core G is the main checkpoint, with L and B models distilled from it. + +| Scale | Tower | Params | Width | Depth | MLP | Heads | CLIP Dim | Resolution / Context Len | +|:-----:|:------:|:------:|:-----:|:-----:|:----:|:-----:|:--------:|:-------------------------:| +| **T/16** | Vision | 0.01B | 192 | 12 | 768 | 3 | 512 | 384px | +| | Text | 0.04B | 512 | 12 | 2048 | 8 | 512 | 32 tokens | +| **S/16** | Vision | 0.02B | 394 | 12 | 1536 | 6 | 512 | 384px | +| | Text | 0.04B | 512 | 12 | 2048 | 8 | 512 | 32 tokens | +| **B/16** | Vision | 0.09B | 768 | 12 | 3072 | 12 | 1024 | 224px | +| | Text | 0.31B | 1024 | 24 | 4096 | 16 | 1024 | 32 tokens | +| **L/14** | Vision | 0.32B | 1024 | 24 | 4096 | 16 | 1024 | 336px | +| | Text | 0.31B | 1024 | 24 | 4096 | 16 | 1024 | 32 tokens | +| **G/14** | Vision | 1.88B | 1536 | 50 | 8960 | 16 | 1280 | 448px | +| | Text | 0.47B | 1280 | 24 | 5120 | 20 | 1280 | 72 tokens | + +All PE core models use an attention pooling block with 8 heads on top of the vision tower. The L and B models _additionally_ have a class token for global aggregation. See the paper for more details. + + + +#### Model Performance +PE core obtains extremely strong results across the board on zero-shot image classification and retrieval _as well as_ zero-shot video classification and retrieval. We present a sample of its performance across those domains below. + +| | Model | Checkpoint | IN-1k | IN-v2 | IN-A | ObjectNet | COCO-T2I | Kinetics-400 | VTT-T2V +|:--:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| +| | **T/16** 384px | [PE-Core-T16-384](https://huggingface.co/facebook/PE-Core-T16-384) | 62.1 | 54.7 | 21.1 | 43.9 | 33.0 | 41.5 | 28.8 | +| | **S/16** 384px | [PE-Core-S16-384](https://huggingface.co/facebook/PE-Core-S16-384) | 72.7 | 65.0 | 49.5 | 60.0 | 42.6 | 55.0 | 39.3 | +| | **B/16** 224px | [PE-Core-B16-224](https://huggingface.co/facebook/PE-Core-B16-224) | 78.4 | 71.7 | 62.4 | 71.9 | 50.9 | 65.6 | 47.6 | +| | **L/14** 336px | [PE-Core-L14-336](https://huggingface.co/facebook/PE-Core-L14-336) | 83.5 | 77.9 | 89.0 | 84.7 | 57.1 | 73.4 | 50.3 | +| | **G/14** 448px | [PE-Core-G14-448](https://huggingface.co/facebook/PE-Core-G14-448) | 85.4 | 80.2 | 92.6 | 88.2 | 58.1 | 76.9 | 51.2 | + +PE core performs particularly well on the _hard_ benchmarks such as ObjectNet and ImageNet-A. + + +## Perception Encoder: Language +PE lang takes the strong language performance from the intermediate layers of PE core and aligns it to the end for use with large language models. We specifically tuned PE lang to be versatile for any multimodal langugage modeling use case, including using different language model decoders (e.g., Llama / Qwen) and using different eval settings (e.g., native res / tiling). PE lang performs particularly well on OCR and document tasks. + +We release two PE Lang checkpoints. Here are their results benchmarked in the frozen encoder [PLM-8B](../plm/README.md) benchmark SFT using 448px _only_ (i.e., _with no tiling_) and Llama 3.1 8B as the decoder: + +| | Encoder | Checkpoint | Doc VQA (val) | InfoQA (val) | TextVQA | MVBench | PerceptionTest (val) | EgoSchema (val) | +|:--:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| +| | **L/14** 448px | [PE-Lang-L14-448](https://huggingface.co/facebook/PE-Lang-L14-448) | 81.9 | 46.4 | 73.0 | 52.3 | 54.7 | 59.8 | +| | **G/14** 448px | [PE-Lang-G14-448](https://huggingface.co/facebook/PE-Lang-G14-448) | 84.4 | 48.3 | 75.2 | 52.4 | 56.0 | 62.0 | + + + +Here is a sample of the performance obtainable by using PE lang G tuned further with [PLM](../plm/README.md) using 36+1 image tiles / 32 video frames and Llama 3.1 as the decoder: + +| | Model | Encoder | Doc VQA (test) | InfoQA (test) | TextVQA | MVBench | PerceptionTest (test) | EgoSchema (test) | +|:--:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| +| | PLM-3B | [PE-Lang-L14-448-Tiling](https://huggingface.co/facebook/PE-Lang-L14-448-Tiling)* | 93.8 | 74.6 | 84.3 | 74.7 | 79.3 | 66.9 | +| | PLM-8B | [PE-Lang-G14-448-Tiling](https://huggingface.co/facebook/PE-Lang-G14-448-Tiling)* | 94.6 | 80.9 | 86.5 | 77.1 | 82.7 | 68.8 | + +\* These checkpoints were aligned with tiling. Use them if you use higher than 448 resolution with tiling in the LLM decoder. + +See the paper for full performance evaluations and fair comparisons to other models. + + + +## Perception Encoder: Spatial +PE spatial similarly takes the strong spatial performance from the intermediate layers of PE core and aligns it to the end using a simple frozen teacher self-distillation loss and further refines with a novel SAM 2.1 mask-based learning strategy. PE spatial performs well on dense prediction tasks such as detection. + +And despite being a short finetuning step using PE core's intermediate layers as a teacher (a pure CLIP model with a global loss) plus a little bit of refinement with SAM, the resulting feature space is quite detailed and well-aligned. Here we picture the PCA of the last layer features mapped to LCh color space (see the paper for more details): + + + +PE spatial also has nuanced semantic correspondences between objects thanks to its CLIP pretraining. Here we show again PCA but only for the tokens not masked. PE spatial shows correspondence between parts like the first image cats' heads, backs, and legs. Additionally, PE spatial can show more nuanced correspondences like for the last two images, where the red/blue directions still denote parts, but the lightness/darkness directions now indicate semantics (i.e., dog/cat breed): + + + +We releease the main PE spatial G checkpoint and several smaller models distilled from it. + +🦾 Main model: +| | Encoder | Checkpoint | ADE20k
[Segmentation](https://github.com/open-mmlab/mmsegmentation)
Linear Probe mIoU | DAVIS
[Tracking](https://github.com/facebookresearch/dino/blob/main/eval_video_segmentation.py)
Zero-Shot J&F | LVIS
[Mask R-CNN](../detection/detectron2_pe/) 1024px
Box / Mask mAP | COCO
[DETA](../detection/DETA_pe/) 1824px
Box mAP | +|:--:|:---:|:---:|:---:|:---:|:---:|:---:| +| | **G/14** 448px | [PE-Spatial-G14-448](https://huggingface.co/facebook/PE-Spatial-G14-448) | 49.3 | 61.5 | 54.2 / 49.3 | 66.0 | + +See the paper for more details and benchmarks for the G model. + +⚗️ Distilled Models: +| | Encoder
(Distilled from G) | Checkpoint | ADE20k
[Segmentation](https://github.com/open-mmlab/mmsegmentation)
Linear Probe mIoU | DAVIS
[Tracking](https://github.com/facebookresearch/dino/blob/main/eval_video_segmentation.py)
Zero-Shot J&F | +|:--:|:---:|:---:|:---:|:---:| +| | **T/16** 512px | [PE-Spatial-T16-512](https://huggingface.co/facebook/PE-Spatial-T16-512) | 27.6 | 55.0 | +| | **S/16** 512px | [PE-Spatial-S16-512](https://huggingface.co/facebook/PE-Spatial-S16-512) | 37.5 | 57.5 | +| | **B/16** 512px | [PE-Spatial-B16-512](https://huggingface.co/facebook/PE-Spatial-B16-512) | 44.4 | 58.9 | +| | **L/14** 448px | [PE-Spatial-L14-448](https://huggingface.co/facebook/PE-Spatial-L14-448) | 48.1 | 60.6 | + +The smaller models are distilled from the G/14 model using a mixturre of pairwise token cosine similarly and direct distillation. More details and evals for the smaller models are coming soon! + + +## Perception Encoder: AudioVisual +Perception Encoder Audiovisual (PE-AV), is the engine behind [SAM Audio](https://ai.meta.com/blog/sam-audio/). It powers core components such as the primary captioning model and SAM Audio Judge, the automatic judge model for audio separation. PE-AV scales up contrastive learning with a robust audiovisual data engine. PE-AV establishes a new state-of-the-art across a wide range of audio and video benchmarks by enabling unified audio-visual-text embeddings. + +#### Audio-Visual Benchmarks + +| | Model | Checkpoint | Avg Retrieval | AudioCaps T→A | AudioCaps T→V | AudioCaps V→A | Clotho T→A | Valor T→A | Valor T→V | VCTK A→T | VGGSound V→A | Internal V→A | +|:--:|:-----:|--------------|---------------|---------------|---------------|---------------|------------|-----------|-----------|----------|---------------|---------------| +| 🆕 | **AV S** 16 frames | [`pe-av-small-16-frame`](https://huggingface.co/facebook/pe-av-small-16-frame) | 45.2 | 41.2 | 18.6 | 75.4 | 24.0 | 29.8 | 70.1 | 96.1 | 34.1 | 17.9 | +| 🆕 | **AV B** 16 frames | [`pe-av-base-16-frame`](https://huggingface.co/facebook/pe-av-base-16-frame) | 47.0 | 43.1 | 19.8 | 80.6 | 23.4 | 31.9 | 70.0 | 94.8 | 39.0 | 20.4 | +| 🆕 | **AV L** 16 frames | [`pe-av-large-16-frame`](https://huggingface.co/facebook/pe-av-large-16-frame) | 48.2 | 44.7 | 19.5 | 86.1 | 22.8 | 35.0 | 70.9 | 85.6 | 45.2 | 23.9 | +| 🆕 | **AV S** all frames | [`pe-av-small`](https://huggingface.co/facebook/pe-av-small) | 48.1 | 41.8 | 18.8 | 77.4 | 23.9 | 29.3 | 70.9 | 94.9 | 35.4 | 40.5 | +| 🆕 | **AV B** all frames | [`pe-av-base`](https://huggingface.co/facebook/pe-av-base) | 50.2 | 42.7 | 19.6 | 83.7 | 23.8 | 30.8 | 71.2 | 94.9 | 40.7 | 44.6 | +| 🆕 | **AV L** all frames | [`pe-av-large`](https://huggingface.co/facebook/pe-av-large) | 51.6 | 45.8 | 20.8 | 88.3 | 23.0 | 35.1 | 70.9 | 85.6 | 48.3 | 46.5 | + + +#### Audio Event Localization Benchmarks + +| | Model | Checkpoint | Internal Bench (AUROC) | ASFX-SED (AUROC) | AudioSet-Strong (AUROC) | DESED (AUROC) | UrbanSED (AUROC) | +|:--:|:-----:|------------------|---------------------|------------------|-----------------------|-------------|-------------| +| 🆕 | **A-Frame S** | [`pe-a-frame-small`](https://huggingface.co/facebook/pe-a-frame-small)| 0.91 | 0.83 | 0.96 | 0.96 | 0.88 | +| 🆕 | **A-Frame B** | [`pe-a-frame-base`](https://huggingface.co/facebook/pe-a-frame-base)| 0.92 | 0.83 | 0.96 | 0.98 | 0.89 | +| 🆕 | **A-Frame L** | [`pe-a-frame-large`](https://huggingface.co/facebook/pe-a-frame-large)| 0.91 | 0.83 | 0.96 | 0.97 | 0.89 | + + +## PE Video Dataset (PVD) +In the process of developing the video data engine we use for PE core, we have collected a high-quality video dataset that contains 1M diverse videos with high visual fidelity and large resolution, split into 10 high level categories. We also annotated 120K samples with the highest amount of motiotion with our video captioning data engine and further asked human annotators to refine the captions. You can find more information about and download PVD [here](https://ai.meta.com/datasets/pe-video/). + + +# Usage + +## Installation +See the installation instructions for `perception_models` in the parent repository: https://github.com/facebookresearch/perception_models.git + +Then download a model using one of the above checkpoint links. + +## Examples +Here are some examples about how to use the models. More coming soon! + +### 1. PE core CLIP Image / Text Feature Extraction +Perception Encoder follows the same structure as [open_clip](https://github.com/mlfoundations/open_clip). You can use the following example for image and language feature extraction. + +```python +import torch +from PIL import Image +import core.vision_encoder.pe as pe +import core.vision_encoder.transforms as transforms + +print("CLIP configs:", pe.CLIP.available_configs()) +# CLIP configs: ['PE-Core-G14-448', 'PE-Core-L14-336', 'PE-Core-B16-224', 'PE-Core-S16-384', 'PE-Core-T16-384'] + +model = pe.CLIP.from_config("PE-Core-L14-336", pretrained=True) # Downloads from HF +model = model.cuda() + +preprocess = transforms.get_image_transform(model.image_size) +tokenizer = transforms.get_text_tokenizer(model.context_length) + +image = preprocess(Image.open("docs/assets/cat.png")).unsqueeze(0).cuda() +text = tokenizer(["a diagram", "a dog", "a cat"]).cuda() + +with torch.no_grad(), torch.autocast("cuda"): + image_features, text_features, logit_scale = model(image, text) + text_probs = (logit_scale * image_features @ text_features.T).softmax(dim=-1) + +print("Label probs:", text_probs) # prints: [[0.0, 0.0, 1.0]] +``` +For a in-depth demo for image and video feature extraction, please refer to our [demo notebook](docs/pe_demo.ipynb). + + + +### 2. Clipbench Evaluation +Please refer to [`docs/evaluation.md`](docs/evaluation.md) for the following benchmarks: +- zero-shot image classifcation +- zero-shot image retrieval +- zero-shot video classifcation +- zero-shot video retrieval + + +### 3. Loading PE core / PE lang / PE spatial Vision Encoder Checkpoints +Loading the vision encoders for PE core, PE lang, and PE spatial for downstream use is similar to the CLIP checkpoints, just using `VisionTransformer` instead. Here you can additionally load PE lang and PE spatial for downstream feature encoding. +```python +import torch +from PIL import Image +import core.vision_encoder.pe as pe +import core.vision_encoder.transforms as transforms + +print("PE configs:", pe.VisionTransformer.available_configs()) +# PE configs: ['PE-Core-G14-448', 'PE-Core-L14-336', 'PE-Core-B16-224', 'PE-Core-S16-384', 'PE-Core-T16-384', 'PE-Lang-G14-448', 'PE-Lang-L14-448', 'PE-Lang-G14-448-Tiling', 'PE-Lang-L14-448-Tiling', 'PE-Spatial-G14-448', 'PE-Spatial-L14-448', 'PE-Spatial-B16-512', 'PE-Spatial-S16-512', 'PE-Spatial-T16-512'] + +model = pe.VisionTransformer.from_config("PE-Lang-L14-448", pretrained=True) # Loads from HF +model = model.cuda() + +preprocess = transforms.get_image_transform(model.image_size) +image = preprocess(Image.open("docs/assets/cat.png")).unsqueeze(0).cuda() + +out = model.forward_features(image, strip_cls_token=True) # pass layer_idx= to get a specific layer's output! +print(out.shape) +# torch.Size([1, 1024, 1024]) +``` + +### 4. PE-AV embedding + +```python +import os +from core.audio_visual_encoder import PEAudioVisual, PEAudioVisualTransform +import torch + +device = torch.device("cuda" if torch.cuda.is_available() else "cpu") +model = PEAudioVisual.from_config("pe-av-large", pretrained=True).to(device) +transform = PEAudioVisualTransform.from_config("pe-av-large") + +video_files = ["assets/train.mp4", "assets/office.mp4"] +descriptions = [ + "A person talking with sirens and a train in the background", + "Two people talking in an office, with sounds of workers typing on a keyboard" +] + +def embed(videos=None, audio=None, text=None): + inputs = transform(videos=videos, audio=audio, text=text) + inputs = inputs.to(device) + with torch.inference_mode(), torch.autocast(device.type, dtype=torch.bfloat16): + return model(**inputs) + +vt_outputs = embed(videos=video_files, text=descriptions) +avt_outputs = embed(videos=video_files, audio=video_files, text=descriptions) +at_outputs = embed(audio=video_files, text=descriptions) + +# Compute dot product between visual and text +vt_dot_products = torch.einsum("ij,ij->i", vt_outputs.visual_embeds, vt_outputs.visual_text_embeds) +# Compute dot product between audio_visual and text +avt_dot_products = torch.einsum("ij,ij->i", avt_outputs.audio_visual_embeds, avt_outputs.audio_visual_text_embeds) +# Compute dot product between audio and text +at_dot_products = torch.einsum("ij,ij->i", at_outputs.audio_embeds, at_outputs.audio_text_embeds) +# Compute dot product between audio and video +av_dot_products = torch.einsum("ij,ij->i", avt_outputs.audio_embeds, avt_outputs.video_embeds) +``` + +### 5. PE-A Frame audio event localization + +```python +from core.audio_visual_encoder import ( + PEAudioFrame, + PEAudioFrameTransform, +) +import torch + +device = torch.device("cuda" if torch.cuda.is_available() else "cpu") +model = PEAudioFrame.from_config("pe-a-frame-large", pretrained=True).to(device) +transform = PEAudioFrameTransform.from_config("pe-a-frame-large") + +descriptions = ["a person talking"] +inputs = transform( + audio=["assets/office.mp4"], + text=descriptions, +).to(device) + +with torch.inference_mode(): + outputs = model(**inputs) + +# Print the spans for each description (start and end timestamps for when they occur in the audio) +for description, spans in zip(descriptions, outputs.spans): + span_str = ", ".join([f"({start:.2f}, {end:.2f})" for start, end in spans]) + print(f'"{description}": [{span_str}]') + +``` + +## Acknowledgement 🙏 +We are thankful to [Open_CLIP](https://github.com/mlfoundations/open_clip) for open-source contributions in CLIP training, and [CLIP_benchmark](https://github.com/LAION-AI/CLIP_benchmark) for CLIP model inference and evaluation. The PE code structure and implementation follow Open_CLIP, and this evaluation is based on CLIP_benchmark. + + +## License +All checkpoints released on this page, unless otherwise specified, are released with the [Apache 2.0 license](https://opensource.org/license/apache-2-0). The code itself is licensed under the parent license of this repository. + +## Citation +```BibTeX +@article{bolya2025PerceptionEncoder, + title={Perception Encoder: The best visual embeddings are not at the output of the network}, + author={Daniel Bolya and Po-Yao Huang and Peize Sun and Jang Hyun Cho and Andrea Madotto and Chen Wei and Tengyu Ma and Jiale Zhi and Jathushan Rajasegaran and Hanoona Rasheed and Junke Wang and Marco Monteiro and Hu Xu and Shiyu Dong and Nikhila Ravi and Daniel Li and Piotr Doll{\'a}r and Christoph Feichtenhofer}, + journal={arXiv:2504.13181}, + year={2025} +} + +@article{cho2025PerceptionLM, + title={PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding}, + author={Jang Hyun Cho and Andrea Madotto and Effrosyni Mavroudi and Triantafyllos Afouras and Tushar Nagarajan and Muhammad Maaz and Yale Song and Tengyu Ma and Shuming Hu and Hanoona Rasheed and Peize Sun and Po-Yao Huang and Daniel Bolya and Suyog Jain and Miguel Martin and Huiyu Wang and Nikhila Ravi and Shashank Jain and Temmy Stark and Shane Moon and Babak Damavandi and Vivian Lee and Andrew Westbury and Salman Khan and Philipp Kr\"{a}henb\"{u}hl and Piotr Doll{\'a}r and Lorenzo Torresani and Kristen Grauman and Christoph Feichtenhofer}, + journal={arXiv:2504.13180}, + year={2025} +} +``` diff --git a/perception_models/apps/pe/clip_benchmark/__init__.py b/perception_models/apps/pe/clip_benchmark/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..883377a4d48ac85243599e4ea9e2a0edaf851e82 --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/__init__.py @@ -0,0 +1,5 @@ +"""Top-level package for CLIP Benchmark.""" + +__author__ = """Mehdi Cherti""" +__email__ = "mehdicherti@gmail.com" +__version__ = "0.1.0" diff --git a/perception_models/apps/pe/clip_benchmark/cli.py b/perception_models/apps/pe/clip_benchmark/cli.py new file mode 100644 index 0000000000000000000000000000000000000000..7220d347ff03bfc81915f6f8c8427810812bcfa0 --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/cli.py @@ -0,0 +1,818 @@ +"""Console script for clip_benchmark.""" + +import argparse +import ast +import csv +import json +import os +import pathlib +import random +import sys +from copy import copy +from itertools import product +from functools import partial + +# Determine the parent directory of the clip_benchmark package +parent_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), "..")) +# Add the parent directory to sys.path +sys.path.insert(0, parent_dir) + +import torch +from clip_benchmark.datasets.builder import (build_dataset, dataset_collection, + get_dataset_collate_fn, + get_dataset_collection_from_file, + get_dataset_default_task, + is_video_dataset, is_audio_dataset) +from clip_benchmark.metrics import (linear_probe, multiclass_retrieval, + visualization, zeroshot_classification, + zeroshot_retrieval) +from clip_benchmark.model_collection import (get_model_collection_from_file, + model_collection) + + +class ParseKwargs(argparse.Action): + def __call__(self, parser, namespace, values, option_string=None): + # kw = {} + kw = getattr(namespace, self.dest, {}) + for value in values: + key, value = value.split("=") + try: + kw[key] = ast.literal_eval(value) + except ValueError: + kw[key] = str( + value + ) # fallback to string (avoid need to escape on command line) + setattr(namespace, self.dest, kw) + + +from dataclasses import dataclass +import core.vision_encoder.pe as pe +import core.vision_encoder.transforms as transforms +from core.audio_visual_encoder import PEAudioVisual, PEAudioVisualTransform + +@dataclass +class Visualization: + enabled: bool = False + delete_post_ln: bool = False + extract_layer: int = None + attn_pooling: str = None # "K", "V" + attn_rollout: bool = False + + +def get_parser_args(): + parser = argparse.ArgumentParser() + subparsers = parser.add_subparsers() + parser_eval = subparsers.add_parser("eval", help="Evaluate") + parser_eval.add_argument( + "--dataset", + type=str, + default="cifar10", + nargs="+", + help="Dataset(s) to use for the benchmark. Can be the name of a dataset, or a collection name ('vtab', 'vtab+', 'imagenet_robustness', 'retrieval') or path of a text file where each line is a dataset name", + ) + parser_eval.add_argument( + "--dataset_root", + default="root", + type=str, + help="dataset root folder where the datasets are downloaded. Can be in the form of a template depending on dataset name, e.g., --dataset_root='datasets/{dataset}'. This is useful if you evaluate on multiple datasets.", + ) + parser_eval.add_argument( + "--split", type=str, default="test", help="Dataset split to use" + ) + parser_eval.add_argument( + "--test_split", + dest="split", + action="store", + type=str, + default="test", + help="Dataset split to use", + ) + parser_eval.add_argument( + "--train_split", + type=str, + nargs="+", + default="train", + help="Dataset(s) train split names", + ) + mutually_exclusive = parser_eval.add_mutually_exclusive_group() + mutually_exclusive.add_argument( + "--val_split", + default=None, + type=str, + nargs="+", + help="Dataset(s) validation split names. Mutually exclusive with val_proportion.", + ) + mutually_exclusive.add_argument( + "--val_proportion", + default=None, + type=float, + nargs="+", + help="what is the share of the train dataset will be used for validation part, if it doesn't predefined. Mutually exclusive with val_split", + ) + parser_eval.add_argument( + "--model", + type=str, + nargs="+", + default=["ViT-B-32-quickgelu"], + help="Model architecture to use from OpenCLIP", + ) + parser_eval.add_argument( + "--pretrained", + type=str, + nargs="+", + default=["laion400m_e32"], + help="Model checkpoint name to use from OpenCLIP", + ) + parser_eval.add_argument( + "--pretrained_model", + type=str, + default="", + nargs="+", + help="Pre-trained model(s) to use. Can be the full model name where `model` and `pretrained` are comma separated (e.g., --pretrained_model='ViT-B-32-quickgelu,laion400m_e32'), a model collection name ('openai' or 'openclip_base' or 'openclip_multilingual' or 'openclip_all'), or path of a text file where each line is a model fullname where model and pretrained are comma separated (e.g., ViT-B-32-quickgelu,laion400m_e32). --model and --pretrained are ignored if --pretrained_model is used.", + ) + parser_eval.add_argument( + "--task", + type=str, + default="auto", + choices=[ + "zeroshot_classification", + "zeroshot_retrieval", + "multiclass_retreival", + "linear_probe", + "auto", + ], + help="Task to evaluate on. With --task=auto, the task is automatically inferred from the dataset.", + ) + parser_eval.add_argument( + "--no_amp", + action="store_false", + dest="amp", + default=True, + help="whether to use mixed precision", + ) + parser_eval.add_argument("--num_workers", default=4, type=int) + parser_eval.add_argument( + "--recall_k", + default=[1, 5, 10], + type=int, + help="for retrieval, select the k for Recall@K metric. ", + nargs="+", + ) + parser_eval.add_argument( + "--fewshot_k", + default=-1, + type=int, + help="for linear probe, how many shots. -1 = whole dataset.", + ) + parser_eval.add_argument( + "--fewshot_epochs", + default=10, + type=int, + help="for linear probe, how many epochs.", + ) + parser_eval.add_argument( + "--fewshot_lr", + default=0.1, + type=float, + help="for linear probe, what is the learning rate.", + ) + parser_eval.add_argument( + "--skip_load", + action="store_true", + help="for linear probes, when everything is cached, no need to load model.", + ) + parser_eval.add_argument( + "--distributed", action="store_true", help="evaluation in parallel" + ) + parser_eval.add_argument("--seed", default=0, type=int, help="random seed.") + parser_eval.add_argument("--batch_size", default=64, type=int) + parser_eval.add_argument( + "--normalize", default=True, type=bool, help="features normalization" + ) + parser_eval.add_argument( + "--model_cache_dir", + default=None, + type=str, + help="directory to where downloaded models are cached", + ) + parser_eval.add_argument( + "--feature_root", + default="features", + type=str, + help="feature root folder where the features are stored.", + ) + parser_eval.add_argument( + "--annotation_file", + default="", + type=str, + help="text annotation file for retrieval datasets. Only needed for when `--task` is `zeroshot_retrieval`.", + ) + parser_eval.add_argument( + "--custom_classname_file", + default=None, + type=str, + help="use custom json file with classnames for each dataset, where keys are dataset names and values are list of classnames.", + ) + parser_eval.add_argument( + "--custom_template_file", + default=None, + type=str, + help="use custom json file with prompts for each dataset, where keys are dataset names and values are list of prompts. For instance, to use CuPL prompts, use --custom_template_file='cupl_prompts.json'", + ) + parser_eval.add_argument( + "--dump_classnames", + default=False, + action="store_true", + help="dump classnames to the results json file.", + ) + parser_eval.add_argument( + "--dump_templates", + default=False, + action="store_true", + help="dump templates to the results json file.", + ) + + parser_eval.add_argument( + "--name", default=None, type=str, help="Overwrite the name used." + ) + + parser_eval.add_argument( + "--image-mean", + type=float, + nargs="+", + default=None, + metavar="MEAN", + help="Override default image mean value of dataset", + ) + parser_eval.add_argument( + "--image-std", + type=float, + nargs="+", + default=None, + metavar="STD", + help="Override default image std deviation of of dataset", + ) + parser_eval.add_argument( + "--force-preprocess-cfg", nargs="*", default={}, action=ParseKwargs + ) + parser_eval.add_argument( + "--force-vision-cfg", + nargs="*", + default={}, + action=ParseKwargs, + help="Overwrite fields of the vision cfg with the args. Only specified kwdargs will be updated.", + ) + parser_eval.add_argument( + "--num-frames", + default=8, + type=int, + help="number of frames to use for video datasets", + ) + parser_eval.add_argument( + "--reweight-retrieval", + default=True, + action=argparse.BooleanOptionalAction, + help="use the softmax trick to reweight the retrieval scores", + ) + parser_eval.add_argument( + "--reweight-scale", + default=1.0, + type=float, + help="Scale the scores prior to doing the softmax trick to reweight" + ) + parser_eval.add_argument( + "--visualize", + nargs="*", + default={}, + action=ParseKwargs, + help="Visualization tool. Spits out visualizations as just bytes in the console. To be read by bento.", + ) + + parser_eval.add_argument( + "--language", + default="en", + type=str, + nargs="+", + help="language(s) of classname and prompts to use for zeroshot classification.", + ) + parser_eval.add_argument( + "--output", + default="result.json", + type=str, + help="output file where to dump the metrics. Can be in form of a template, e.g., --output='{dataset}_{pretrained}_{model}_{language}_{task}.json'", + ) + parser_eval.add_argument( + "--quiet", + dest="verbose", + action="store_false", + help="suppress verbose messages", + ) + parser_eval.add_argument( + "--save_clf", + default=None, + type=str, + help="optionally save the classification layer output by the text tower", + ) + parser_eval.add_argument( + "--load_clfs", + nargs="+", + default=[], + type=str, + help="optionally load and average mutliple layers output by text towers.", + ) + parser_eval.add_argument( + "--skip_existing", + default=False, + action="store_true", + help="whether to skip an evaluation if the output file exists.", + ) + parser_eval.add_argument( + "--model_type", default="open_clip", type=str, help="clip model type" + ) + parser_eval.add_argument( + "--wds_cache_dir", + default=None, + type=str, + help="optional cache directory for webdataset only", + ) + parser_eval.set_defaults(which="eval") + + parser_build = subparsers.add_parser("build", help="Build CSV from evaluations") + parser_build.add_argument( + "files", type=str, nargs="+", help="path(s) of JSON result files" + ) + parser_build.add_argument( + "--output", type=str, default="benchmark.csv", help="CSV output file" + ) + parser_build.set_defaults(which="build") + + args = parser.parse_args() + return parser, args + + +def main(): + parser, base = get_parser_args() + if not hasattr(base, "which"): + parser.print_help() + return + if base.which == "eval": + main_eval(base) + elif base.which == "build": + main_build(base) + + +def main_build(base): + # Build a benchmark single CSV file from a set of evaluations (JSON files) + rows = [] + fieldnames = set() + + def process_file(path: str): + data = json.load(open(path)) + row = {} + row.update(data["metrics"]) + row.update(data) + del row["metrics"] + row["model_fullname"] = row["model"] + " " + row["pretrained"] + for field in row.keys(): + fieldnames.add(field) + rows.append(row) + + for path in base.files: + if os.path.isdir(path): + files = [ + os.path.join(path, f) for f in os.listdir(path) if f.endswith(".json") + ] + for file in files: + process_file(file) + else: + process_file(path) + with open(base.output, "w") as csvfile: + writer = csv.DictWriter(csvfile, fieldnames=fieldnames) + writer.writeheader() + for row in rows: + writer.writerow(row) + + +def main_eval(base): + # Get list of pre-trained models to evaluate + pretrained_model = _as_list(base.pretrained_model) + if pretrained_model: + models = [] + for name in pretrained_model: + if os.path.isfile(name): + # if path, read file, each line is a pre-trained model + models.extend(get_model_collection_from_file(name)) + elif name in model_collection: + # if part of `model_collection`, retrieve from it + models.extend(model_collection[name]) + else: + # if not, assume it is in the form of `model,pretrained` + model, pretrained = name.split(",") + models.append((model, pretrained)) + else: + models = list(product(base.model, base.pretrained)) + + # Get list of datasets to evaluate on + datasets = [] + for name in _as_list(base.dataset): + if os.path.isfile(name): + # If path, read file, each line is a dataset name + datasets.extend(get_dataset_collection_from_file(name)) + elif name in dataset_collection: + # if part of `dataset_collection`, retrieve from it + datasets.extend(dataset_collection[name]) + else: + # if not, assume it is simply the name of the dataset + datasets.append(name) + train_splits = _as_list(base.train_split) + train_splits = _single_option_to_multiple_datasets( + train_splits, datasets, "train_split" + ) + proportions, val_splits = None, None + if base.val_split is not None: + val_splits = _as_list(base.val_split) + val_splits = _single_option_to_multiple_datasets( + val_splits, datasets, "val_split" + ) + if base.val_proportion is not None: + proportions = _as_list(base.val_proportion) + proportions = _single_option_to_multiple_datasets( + proportions, datasets, "val_proportion" + ) + + dataset_info = {} + for i in range(len(datasets)): + dataset_info[datasets[i]] = { + "train_split": train_splits[i], + "val_split": val_splits[i] if val_splits is not None else None, + "proportion": proportions[i] if proportions is not None else None, + } + # Get list of languages to evaluate on + languages = _as_list(base.language) + + if base.verbose: + print(f"Models: {models}") + print(f"Datasets: {datasets}") + print(f"Languages: {languages}") + runs = product(models, datasets, languages) + if base.distributed: + local_rank, rank, world_size = world_info_from_env() + runs = list(runs) + # randomize runs so that runs are balanced across gpus + random.seed(base.seed) + random.shuffle(runs) + runs = [r for i, r in enumerate(runs) if i % world_size == rank] + for (model, pretrained), (dataset), (language) in runs: + # We iterative over all possible model/dataset/languages + args = copy(base) + args.model = model + args.pretrained = pretrained + args.dataset = dataset + args.language = language + args.train_split = dataset_info[dataset]["train_split"] + args.val_split = dataset_info[dataset]["val_split"] + args.val_proportion = dataset_info[dataset]["proportion"] + run(args) + + +def _as_list(l): + if not l: + return [] + return [l] if type(l) != list else l + + +def _single_option_to_multiple_datasets(cur_option, datasets, name): + cur_len = len(cur_option) + ds_len = len(datasets) + if cur_len != ds_len: + # If user wants to use same value for all datasets + if cur_len == 1: + return [cur_option[0]] * ds_len + else: + raise ValueError(f"The incommensurable number of {name}") + else: + return cur_option + + +def get_basename_and_parent_folder(path): + """ + Returns the basename and the folder two parents above. + Args: + path (str): The input path. + Returns: + str: The basename and the folder two parents above. + """ + p = pathlib.Path(path) + parent_folder = ( + p.parents[1].name if len(p.parents) >= 2 else "" + ) # Get the name of the parent folder two levels up + basename = p.stem # Get the basename + return f"{parent_folder}-{basename}" + + +def run(args): + """Console script for clip_benchmark.""" + if torch.cuda.is_available(): + if args.distributed: + local_rank, rank, world_size = world_info_from_env() + device = "cuda:%d" % local_rank + torch.cuda.set_device(device) + else: + device = "cuda" + args.device = device + else: + args.device = "cpu" + # set seed. + torch.manual_seed(args.seed) + task = args.task + if args.dataset.startswith("wds/"): + dataset_name = args.dataset.replace("wds/", "", 1) + else: + dataset_name = args.dataset + if task == "auto": + task = get_dataset_default_task(dataset_name) + local_model_flag = os.path.isfile(args.pretrained) or os.path.isdir(args.pretrained) + pretrained_slug = ( + get_basename_and_parent_folder(args.pretrained) + if local_model_flag + else args.pretrained + ) + pretrained_slug_full_path = ( + args.pretrained.replace("/", "_") + if os.path.isfile(args.pretrained) + else args.pretrained + ) + dataset_slug = dataset_name.replace("/", "_") + output = args.output.format( + model=args.model, + pretrained=( + pretrained_slug if args.name is None else f"{pretrained_slug}-{args.name}" + ), + pretrained_full_path=pretrained_slug_full_path, + task=task, + dataset=dataset_slug, + language=args.language, + num_frames=args.num_frames, + ) + # hack to replace timm/hf model + output = output.replace("timm/", "timm_") + + if os.path.exists(output) and args.skip_existing: + if args.verbose: + print(f"Skip {output}, exists already.") + return + if args.verbose: + print( + f"Running '{task}' on '{dataset_name}' with the model '{args.pretrained}' on language '{args.language}'" + ) + dataset_root = args.dataset_root.format( + dataset=dataset_name, dataset_cleaned=dataset_name.replace("/", "-") + ) + if args.skip_load: + model, transform, collate_fn, dataloader = None, None, None, None + else: + if args.model.startswith("pe-av"): + # Load PE-AV model + model = PEAudioVisual.from_config(args.model, pretrained=True).cuda() + transform = PEAudioVisualTransform.from_config(args.model) + tokenizer = partial(transform.tokenizer, padding=True, return_tensors="pt") + else: + model_name = args.model + model = pe.CLIP.from_config(model_name, pretrained=True) # Downloads from HF + model = model.cuda() + transform = transforms.get_image_transform(model.image_size) + tokenizer = transforms.get_text_tokenizer(model.context_length) + + model.eval() + + dataset = build_dataset( + dataset_name=args.dataset, + root=dataset_root, + transform=transform, + split=args.split, + annotation_file=args.annotation_file, + download=True, + language=args.language, + task=task, + custom_template_file=args.custom_template_file, + custom_classname_file=args.custom_classname_file, + wds_cache_dir=args.wds_cache_dir, + num_frames=args.num_frames, + ) + if hasattr(dataset, "collate_fn"): + collate_fn = dataset.collate_fn + else: + collate_fn = get_dataset_collate_fn(args.dataset) + + if hasattr(transform, "collate_fn"): + # Union the dataloader's collate fn (which deals with images) with clipbench's collate fn (which deals with text) + def collate_union(collate_img, collate_text): + def collate(batch): + img_batch = collate_img([(x[0], torch.zeros(1)) for x in batch]) + text_batch = collate_text( + [[torch.zeros(3, 224, 224)] + list(x[1:]) for x in batch] + ) + return tuple(list(img_batch[:1]) + list(text_batch[1:])) + + return collate + + collate_fn = collate_union(transform.collate_fn, collate_fn) + + def test(*args, **kwargs): + breakpoint() + return collate_fn(*args, **kwargs) + + if args.verbose: + try: + print(f"Dataset size: {len(dataset)}") + except TypeError: + print("IterableDataset has no len()") + print(f"Dataset split: {args.split}") + if hasattr(dataset, "classes") and dataset.classes: + try: + print(f"Dataset classes: {dataset.classes}") + print(f"Dataset number of classes: {len(dataset.classes)}") + except AttributeError: + print("Dataset has no classes.") + + if args.dataset.startswith("wds/"): + dataloader = torch.utils.data.DataLoader( + dataset.batched(args.batch_size, collation_fn=collate_fn), + batch_size=None, + shuffle=False, + num_workers=args.num_workers, + ) + else: + dataloader = torch.utils.data.DataLoader( + dataset, + batch_size=args.batch_size, + shuffle=False, + num_workers=args.num_workers, + collate_fn=collate_fn, + ) + + + if task == "zeroshot_classification": + zeroshot_templates = ( + dataset.templates if hasattr(dataset, "templates") else None + ) + if args.verbose: + print(f"Zero-shot templates: {zeroshot_templates}") + classnames = dataset.classes if hasattr(dataset, "classes") else None + assert ( + zeroshot_templates is not None and classnames is not None + ), "Dataset does not support classification" + metrics = zeroshot_classification.evaluate( + model, + dataloader, + tokenizer, + classnames, + zeroshot_templates, + video_dataset=is_video_dataset(args.dataset), + device=args.device, + amp=args.amp, + verbose=args.verbose, + save_clf=args.save_clf, + load_clfs=args.load_clfs, + args=args, + ) + elif task == "zeroshot_retrieval": + metrics = zeroshot_retrieval.evaluate( + model, + dataloader, + tokenizer, + video_dataset=is_video_dataset(args.dataset), + recall_k_list=args.recall_k, + device=args.device, + amp=args.amp, + args=args, + audio_dataset=is_audio_dataset(args.dataset), + transform=transform, + ) + elif task == "multiclass_retrieval": + metrics = multiclass_retrieval.evaluate( + model, + dataloader, + tokenizer, + device=args.device, + amp=args.amp, + args=args, + retrieval_template=dataset.retrieval_template, + ) + elif task == "linear_probe": + # we also need the train and validation splits for linear probing. + train_dataset = None + train_dataset = build_dataset( + dataset_name=args.dataset, + root=dataset_root, + transform=transform, + split=args.train_split, + annotation_file=args.annotation_file, + download=True, + ) + if args.val_split is not None: + val_dataset = build_dataset( + dataset_name=args.dataset, + root=dataset_root, + transform=transform, + split=args.val_split, + annotation_file=args.annotation_file, + download=True, + ) + elif args.val_proportion is not None: + train_dataset, val_dataset = torch.utils.data.random_split( + train_dataset, [1 - args.val_proportion, args.val_proportion] + ) + else: + val_dataset = None + train_dataloader = torch.utils.data.DataLoader( + train_dataset, + batch_size=args.batch_size, + shuffle=False, + num_workers=args.num_workers, + collate_fn=collate_fn, + pin_memory=True, + ) + if val_dataset is not None: + val_dataloader = torch.utils.data.DataLoader( + val_dataset, + batch_size=args.batch_size, + shuffle=False, + num_workers=args.num_workers, + collate_fn=collate_fn, + pin_memory=True, + ) + else: + val_dataloader = None + metrics = linear_probe.evaluate( + model, + train_dataloader, + dataloader, + args.fewshot_k, + args.batch_size, + args.num_workers, + args.fewshot_lr, + args.fewshot_epochs, + (args.model + "-" + args.pretrained + "-" + args.dataset).replace("/", "_"), + args.seed, + args.feature_root, + val_dataloader=val_dataloader, + device=args.device, + normalize=args.normalize, + amp=args.amp, + verbose=args.verbose, + ) + else: + raise ValueError( + "Unsupported task: {}. task should be `zeroshot_classification`, `zeroshot_retrieval`, `multiclass_retrieval`, `linear_probe`, or `captioning`".format( + task + ) + ) + dump = { + "dataset": args.dataset, + "model": args.model, + "pretrained": args.pretrained, + "task": task, + "metrics": metrics, + "language": args.language, + "name": args.name if args.name is not None else "", + } + if hasattr(dataset, "classes") and dataset.classes and args.dump_classnames: + dump["classnames"] = dataset.classes + if hasattr(dataset, "templates") and dataset.templates and args.dump_templates: + dump["templates"] = dataset.templates + if args.verbose: + print(f"Dump results to: {output}") + os.makedirs(os.path.dirname(output), exist_ok=True) + with open(output, "w") as f: + json.dump(dump, f) + return 0 + + +def world_info_from_env(): + # from openclip + local_rank = 0 + for v in ( + "LOCAL_RANK", + "MPI_LOCALRANKID", + "SLURM_LOCALID", + "OMPI_COMM_WORLD_LOCAL_RANK", + ): + if v in os.environ: + local_rank = int(os.environ[v]) + break + global_rank = 0 + for v in ("RANK", "PMI_RANK", "SLURM_PROCID", "OMPI_COMM_WORLD_RANK"): + if v in os.environ: + global_rank = int(os.environ[v]) + break + world_size = 1 + for v in ("WORLD_SIZE", "PMI_SIZE", "SLURM_NTASKS", "OMPI_COMM_WORLD_SIZE"): + if v in os.environ: + world_size = int(os.environ[v]) + break + return local_rank, global_rank, world_size + + +if __name__ == "__main__": + sys.exit(main()) # pragma: no cover diff --git a/perception_models/apps/pe/clip_benchmark/datasets/__init__.py b/perception_models/apps/pe/clip_benchmark/datasets/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/perception_models/apps/pe/clip_benchmark/datasets/audiocaps.py b/perception_models/apps/pe/clip_benchmark/datasets/audiocaps.py new file mode 100644 index 0000000000000000000000000000000000000000..445bc20915e815297dc3586ea52e0daa8541b206 --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/datasets/audiocaps.py @@ -0,0 +1,63 @@ +import torch +import os +from subprocess import check_call +import pandas +import warnings + +class Audiocaps(torch.utils.data.Dataset): + def __init__(self, transform, root): + self.root = root + self.transform = transform + cache_dir = os.environ.get("PERCEPTION_MODELS_CACHE", os.path.expanduser("~/.cache/perception_encoder")) + cache_dir = os.path.join(cache_dir, "audiocaps") + os.makedirs(cache_dir, exist_ok=True) + outfile = os.path.join(cache_dir, "test.csv") + if not os.path.exists(outfile): + url = "https://raw.githubusercontent.com/cdjkim/audiocaps/refs/heads/master/dataset/test.csv" + check_call(["curl", url, "--output", outfile + ".tmp"]) + os.rename(outfile + ".tmp", outfile) + self.df = pandas.read_csv(outfile) + self.df = self.df.groupby(["youtube_id", "start_time"])["caption"].agg(list).reset_index() + + audio_files, video_files = [], [] + + for yt_id in self.df["youtube_id"].values: + audio_file = os.path.join(self.root, "audiocaps/audio", f"{yt_id}.flac") + # Since youtube videos aren't permanent, we allow for some to be missing + if not os.path.exists(audio_file): + warnings.warn(f"Audio file {audio_file} does not exist, skipping...") + audio_file = None + audio_files.append(audio_file) + video_file = os.path.join(self.root, "audiocaps/video", f"{yt_id}.mp4") + if not os.path.exists(video_file): + warnings.warn(f"Video file {video_file} does not exist, skipping...") + video_file = None + video_files.append(video_file) + + self.df["video_file"] = video_files + self.df["audio_file"] = audio_files + self.df = self.df[self.df["audio_file"].notnull() & self.df["video_file"].notnull()] + + def collate_fn(self, batch): + source, target = zip(*batch) + return source, target + + def __len__(self): + return len(self.df) + + +class AudiocapsAudioVideo(Audiocaps): + def __getitem__(self, idx): + row = self.df.iloc[idx] + return row["audio_file"], [row["video_file"]] + +class AudiocapsAudioText(Audiocaps): + def __getitem__(self, idx): + row = self.df.iloc[idx] + return row["audio_file"], row["caption"] + + +class AudiocapsVideoText(Audiocaps): + def __getitem__(self, idx): + row = self.df.iloc[idx] + return row["video_file"], row["caption"] diff --git a/perception_models/apps/pe/clip_benchmark/datasets/babel_imagenet.py b/perception_models/apps/pe/clip_benchmark/datasets/babel_imagenet.py new file mode 100644 index 0000000000000000000000000000000000000000..fd4fd27edf302fde6abed82fb718b1db08b5bb36 --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/datasets/babel_imagenet.py @@ -0,0 +1,28 @@ +import torchvision + +""" +BabelImageNet from https://arxiv.org/pdf/2306.08658.pdf +Adapted from https://github.com/gregor-ge/Babel-ImageNet, thanks to the authors +""" + + +class BabelImageNet(torchvision.datasets.ImageNet): + def __init__( + self, root: str, idxs, split: str = "val", download=None, **kwargs + ) -> None: + super().__init__(root, split, **kwargs) + examples_per_class = len(self.targets) // 1000 + select_idxs = [ + idx * examples_per_class + i + for idx in idxs + for i in range(examples_per_class) + ] + self.targets = [i for i in range(len(idxs)) for _ in range(examples_per_class)] + self.imgs = [self.imgs[i] for i in select_idxs] + self.samples = [self.samples[i] for i in select_idxs] + self.idxs = idxs + + def __getitem__(self, i): + img, target = super().__getitem__(i) + target = self.idxs.index(target) + return img, target diff --git a/perception_models/apps/pe/clip_benchmark/datasets/builder.py b/perception_models/apps/pe/clip_benchmark/datasets/builder.py new file mode 100644 index 0000000000000000000000000000000000000000..df80e769ff6fa1117209dac813495501b15a76d3 --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/datasets/builder.py @@ -0,0 +1,3105 @@ +import json +import os +import sys +import warnings +from subprocess import call + +import torch +from torch.utils.data import default_collate +from torchvision.datasets import (CIFAR10, CIFAR100, DTD, GTSRB, MNIST, PCAM, + STL10, SUN397, CocoCaptions, Country211, + EuroSAT, FGVCAircraft, Flowers102, Food101, + ImageFolder, ImageNet, OxfordIIITPet, + RenderedSST2, StanfordCars) + +from . import (audiocaps, babel_imagenet, caltech101, clotho_v2, flickr, imagenetv2, objectnet, + pos_neg_caption_dataset, video_classification_dataset, + video_retrieval_dataset, voc2007, winoground) + +MSRVTT_ANN = "PATH_TO/MSRVTT_JSFUSION_test.csv" +MSRVTT_DATA = "PATH_TO/msrvtt/videos/all/" +PVD_ANN = "PATH_TO/imago/annotations/imago15k.csv" +PVD_DATA = "PATH_TO/imago/videos/" +MSVD_ANN = "PATH_TO/msvd/msvd_test_multi.csv" +MSVD_DATA = "PATH_TO/msvd/video/YouTubeClips/" +DIDEMO_ANN = "PATH_TO/didemo/didemo_test.csv" +DIDEMO_DATA = "PATH_TO/didemo/all_videos/videos/" +ANET_ANN = "PATH_TO/anet/anet_test_valid.csv" +ANET_DATA = "PATH_TO/anet/videos/" + +K400_ROOT = "PATH_TO/video_datasets/k400" +K600_ROOT = "PATH_TO/video_datasets/k600" +K700_ROOT = "PATH_TO/video_datasets/k700" +UCF_ROOT = "PATH_TO/ucf/videos" +UCF_PROMPT = "PATH_TO/ucf/custom_labels.txt" +HMDB_ROOT = "PATH_TO/hmdb/112018/data" +HMDB_PROMPT = "PATH_TO/hmdb/hmdb.txt" +MITV1_ROOT = "PATH_TO/Multi-Moments/Multi_Moments_in_Time/videos" +SSV2_ROOT = "PATH_TO/SSv2/videos/val_processed/" + + +def build_dataset( + dataset_name, + root="root", + transform=None, + split="test", + download=True, + annotation_file=None, + language="en", + task="zeroshot_classification", + wds_cache_dir=None, + custom_classname_file=None, + custom_template_file=None, + num_frames=8, + **kwargs, +): + """ + Main function to use in order to build a dataset instance, + + dataset_name: str + name of the dataset + + root: str + root folder where the dataset is downloaded and stored. can be shared among datasets. + + transform: torchvision transform applied to images + + split: str + split to use, depending on the dataset can have different options. + In general, `train` and `test` are available. + For specific splits, please look at the corresponding dataset. + + annotation_file: str or None + only for datasets with captions (used for retrieval) such as COCO + and Flickr. + + custom_classname_file: str or None + Custom classname file where keys are dataset names and values are list of classnames. + + custom_template_file: str or None + Custom template file where keys are dataset names and values are list of prompts, or dicts + where keys are classnames and values are class-specific prompts. + + """ + use_classnames_and_templates = task in ("zeroshot_classification", "linear_probe") + if use_classnames_and_templates: # Only load templates and classnames if we have to + current_folder = os.path.dirname(__file__) + + # Load _classnames.json (packaged with CLIP benchmark that are used by default) + default_classname_file = os.path.join( + current_folder, language + "_classnames.json" + ) + if os.path.exists(default_classname_file): + with open(default_classname_file, "r") as f: + default_classnames = json.load(f) + else: + default_classnames = None + + # Load _zeroshot_classification_templates.json (packaged with CLIP benchmark that are used by default) + default_template_file = os.path.join( + current_folder, language + "_zeroshot_classification_templates.json" + ) + if os.path.exists(default_template_file): + with open(default_template_file, "r") as f: + default_templates = json.load(f) + else: + default_templates = None + + # Load custom classnames file if --custom_classname_file is specified + if custom_classname_file: + if not os.path.exists(custom_classname_file): + custom_classname_file = os.path.join( + current_folder, custom_classname_file + ) + assert os.path.exists( + custom_classname_file + ), f"Custom classname file '{custom_classname_file}' does not exist" + with open(custom_classname_file, "r") as f: + custom_classnames = json.load(f) + else: + custom_classnames = None + + # Load custom template file if --custom_template_file is specified + if custom_template_file: + if not os.path.exists(custom_template_file): + # look at current_folder + custom_template_file = os.path.join( + current_folder, custom_template_file + ) + assert os.path.exists( + custom_template_file + ), f"Custom template file '{custom_template_file}' does not exist" + with open(custom_template_file, "r") as f: + custom_templates = json.load(f) + else: + custom_templates = None + + def download_imagenet(r): + os.makedirs(r, exist_ok=True) + call( + f"wget https://image-net.org/data/ILSVRC/2012/ILSVRC2012_devkit_t12.tar.gz --output-document={r}/ILSVRC2012_devkit_t12.tar.gz", + shell=True, + ) + call( + f"wget https://image-net.org/data/ILSVRC/2012/ILSVRC2012_img_val.tar --output-document={r}/ILSVRC2012_img_val.tar", + shell=True, + ) + + train = split == "train" + if dataset_name in video_classification_datasets.keys(): + task_config = video_classification_datasets[dataset_name] + ds = video_classification_dataset.VideoClassificationDataset( + "", task_config, transform, num_frames=num_frames + ) + elif dataset_name == "clotho-v2": + ds = clotho_v2.ClothoV2(transform) + elif dataset_name == "audiocaps-audio-text": + ds = audiocaps.AudiocapsAudioText(transform, root=root) + elif dataset_name == "audiocaps-video-text": + ds = audiocaps.AudiocapsVideoText(transform, root=root) + elif dataset_name == "audiocaps-audio-video": + ds = audiocaps.AudiocapsAudioVideo(transform, root=root) + elif dataset_name == "msrvtt": + ds = video_retrieval_dataset.VideoRetrievalDataset( + MSRVTT_ANN, + MSRVTT_DATA, + transform, + num_frames=num_frames, + ) + elif dataset_name == "imago_video": + ds = video_retrieval_dataset.VideoRetrievalDataset( + PVD_ANN, + PVD_DATA, + transform, + num_frames=num_frames, + ) + elif dataset_name == "msvd": + ds = video_retrieval_dataset.VideoRetrievalDataset( + MSVD_ANN, + MSVD_DATA, + transform, + num_frames=num_frames, + video_ext="avi", + multi_sent=True, + ) + elif dataset_name == "didemo": + ds = video_retrieval_dataset.VideoRetrievalDataset( + DIDEMO_ANN, + DIDEMO_DATA, + transform, + num_frames=num_frames, + ) + elif dataset_name == "anet": + ds = video_retrieval_dataset.VideoRetrievalDataset( + ANET_ANN, + ANET_DATA, + transform, + num_frames=32, + ) + elif dataset_name == "cifar10": + assert split in ( + "train", + "test", + ), f"Only `train` and `test` split available for {dataset_name}" + ds = CIFAR10( + root=root, train=train, transform=transform, download=download, **kwargs + ) + elif dataset_name == "cifar100": + assert split in ( + "train", + "test", + ), f"Only `train` and `test` split available for {dataset_name}" + ds = CIFAR100( + root=root, train=train, transform=transform, download=download, **kwargs + ) + elif dataset_name == "imagenet1k": + assert split in ( + "train", + "test", + ), f"Only `train` and `test` split available for {dataset_name}" + if not os.path.exists(root): + download_imagenet(root) + ds = ImageNet( + root=root, split="train" if train else "val", transform=transform, **kwargs + ) + ds.classes = default_classnames["imagenet1k"] + elif dataset_name == "imagenet-w": + assert split in ( + "train", + "test", + ), f"Only `train` and `test` split available for {dataset_name}" + from imagenet_w import AddWatermark + from torchvision.transforms import CenterCrop, Normalize + + if not os.path.exists(root): + download_imagenet(root) + index_normalize = None + crop_size = None + for i, t in enumerate(transform.transforms): + if isinstance(t, Normalize): + index_normalize = i + elif isinstance(t, CenterCrop): + crop_size = min(t.size) + assert crop_size is not None, "CenterCrop not found in transform" + assert index_normalize is not None, "Normalize not found in transform" + transform.transforms.insert(index_normalize, AddWatermark(crop_size)) + ds = ImageNet( + root=root, split="train" if train else "val", transform=transform, **kwargs + ) + ds.classes = custom_classnames["imagenet1k"] + elif dataset_name == "babel_imagenet": + assert split in ( + "train", + "test", + ), f"Only `train` and `test` split available for {dataset_name}" + # babel ImageNet from https://github.com/gregor-ge/Babel-ImageNet + if not os.path.exists(root): + download_imagenet(root) + classnames = json.load( + open(os.path.join(current_folder, "babel_imagenet.json")) + ) + assert ( + language.upper() in classnames + ), f"Language '{language}' not supported for Babel-ImageNet" + classnames = classnames[language.upper()] + templates = json.load( + open(os.path.join(current_folder, "nllb_dist13b_prompts.json")) + ) + templates = templates[language.upper()] + templates = [t.replace("{}", "{c}") for t in templates] + idxs, classnames = classnames + ds = babel_imagenet.BabelImageNet( + root=root, + idxs=idxs, + split="train" if train else "val", + transform=transform, + **kwargs, + ) + ds.classes = classnames + ds.templates = templates + elif dataset_name == "imagenet1k-unverified": + assert split in ( + "train", + "test", + ), f"Only `train` and `test` split available for {dataset_name}" + split = "train" if train else "val" + ds = ImageFolder(root=os.path.join(root, split), transform=transform, **kwargs) + # use classnames from OpenAI + ds.classes = default_classnames["imagenet1k"] + elif dataset_name == "imagenetv2": + assert split == "test", f"Only `test` split available for {dataset_name}" + os.makedirs(root, exist_ok=True) + ds = imagenetv2.ImageNetV2Dataset( + variant="matched-frequency", transform=transform, location=root + ) + ds.classes = default_classnames["imagenet1k"] + elif dataset_name == "imagenet_sketch": + assert split == "test", f"Only `test` split available for {dataset_name}" + # Downloadable from https://drive.google.com/open?id=1Mj0i5HBthqH1p_yeXzsg22gZduvgoNeA + if not os.path.exists(root): + # Automatic download + print("Downloading imagenet_sketch...") + if not has_gdown(): + print( + "GDown is needed to download the dataset. Please install it via `pip install gdown`" + ) + sys.exit(1) + # Download ImageNet-Sketch.zip + call("gdown --id 1Mj0i5HBthqH1p_yeXzsg22gZduvgoNeA", shell=True) + assert os.path.exists("ImageNet-Sketch.zip") + # Unzip and move to `root` + call("unzip ImageNet-Sketch.zip", shell=True) + call(f"mv sketch {root}", shell=True) + ds = ImageFolder(root=root, transform=transform, **kwargs) + ds.classes = default_classnames["imagenet1k"] + elif dataset_name == "imagenet-a": + assert split == "test", f"Only `test` split available for {dataset_name}" + # Downloadable from https://people.eecs.berkeley.edu/~hendrycks/imagenet-a.tar + if not os.path.exists(root): + print("Downloading imagenet-a...") + call( + "wget https://people.eecs.berkeley.edu/~hendrycks/imagenet-a.tar", + shell=True, + ) + # Untar and move to `root` + call("tar xvf imagenet-a.tar", shell=True) + call(f"mv imagenet-a {root}", shell=True) + ds = ImageFolder(root=root, transform=transform, **kwargs) + ds.classes = default_classnames["imagenet1k"] + imagenet_a_wnids = [ + "n01498041", + "n01531178", + "n01534433", + "n01558993", + "n01580077", + "n01614925", + "n01616318", + "n01631663", + "n01641577", + "n01669191", + "n01677366", + "n01687978", + "n01694178", + "n01698640", + "n01735189", + "n01770081", + "n01770393", + "n01774750", + "n01784675", + "n01819313", + "n01820546", + "n01833805", + "n01843383", + "n01847000", + "n01855672", + "n01882714", + "n01910747", + "n01914609", + "n01924916", + "n01944390", + "n01985128", + "n01986214", + "n02007558", + "n02009912", + "n02037110", + "n02051845", + "n02077923", + "n02085620", + "n02099601", + "n02106550", + "n02106662", + "n02110958", + "n02119022", + "n02123394", + "n02127052", + "n02129165", + "n02133161", + "n02137549", + "n02165456", + "n02174001", + "n02177972", + "n02190166", + "n02206856", + "n02219486", + "n02226429", + "n02231487", + "n02233338", + "n02236044", + "n02259212", + "n02268443", + "n02279972", + "n02280649", + "n02281787", + "n02317335", + "n02325366", + "n02346627", + "n02356798", + "n02361337", + "n02410509", + "n02445715", + "n02454379", + "n02486410", + "n02492035", + "n02504458", + "n02655020", + "n02669723", + "n02672831", + "n02676566", + "n02690373", + "n02701002", + "n02730930", + "n02777292", + "n02782093", + "n02787622", + "n02793495", + "n02797295", + "n02802426", + "n02814860", + "n02815834", + "n02837789", + "n02879718", + "n02883205", + "n02895154", + "n02906734", + "n02948072", + "n02951358", + "n02980441", + "n02992211", + "n02999410", + "n03014705", + "n03026506", + "n03124043", + "n03125729", + "n03187595", + "n03196217", + "n03223299", + "n03250847", + "n03255030", + "n03291819", + "n03325584", + "n03355925", + "n03384352", + "n03388043", + "n03417042", + "n03443371", + "n03444034", + "n03445924", + "n03452741", + "n03483316", + "n03584829", + "n03590841", + "n03594945", + "n03617480", + "n03666591", + "n03670208", + "n03717622", + "n03720891", + "n03721384", + "n03724870", + "n03775071", + "n03788195", + "n03804744", + "n03837869", + "n03840681", + "n03854065", + "n03888257", + "n03891332", + "n03935335", + "n03982430", + "n04019541", + "n04033901", + "n04039381", + "n04067472", + "n04086273", + "n04099969", + "n04118538", + "n04131690", + "n04133789", + "n04141076", + "n04146614", + "n04147183", + "n04179913", + "n04208210", + "n04235860", + "n04252077", + "n04252225", + "n04254120", + "n04270147", + "n04275548", + "n04310018", + "n04317175", + "n04344873", + "n04347754", + "n04355338", + "n04366367", + "n04376876", + "n04389033", + "n04399382", + "n04442312", + "n04456115", + "n04482393", + "n04507155", + "n04509417", + "n04532670", + "n04540053", + "n04554684", + "n04562935", + "n04591713", + "n04606251", + "n07583066", + "n07695742", + "n07697313", + "n07697537", + "n07714990", + "n07718472", + "n07720875", + "n07734744", + "n07749582", + "n07753592", + "n07760859", + "n07768694", + "n07831146", + "n09229709", + "n09246464", + "n09472597", + "n09835506", + "n11879895", + "n12057211", + "n12144580", + "n12267677", + ] + imagenet_a_mask = [ + wnid in set(imagenet_a_wnids) for wnid in all_imagenet_wordnet_ids + ] + ds.classes = [cl for cl, mask in zip(ds.classes, imagenet_a_mask) if mask] + elif dataset_name == "imagenet-r": + assert split == "test", f"Only `test` split available for {dataset_name}" + # downloadable from https://people.eecs.berkeley.edu/~hendrycks/imagenet-r.tar + if not os.path.exists(root): + print("Downloading imagenet-r...") + call( + "wget https://people.eecs.berkeley.edu/~hendrycks/imagenet-r.tar", + shell=True, + ) + # Untar and move to `root` + call("tar xvf imagenet-r.tar", shell=True) + call(f"mv imagenet-r {root}", shell=True) + imagenet_r_wnids = { + "n01443537", + "n01484850", + "n01494475", + "n01498041", + "n01514859", + "n01518878", + "n01531178", + "n01534433", + "n01614925", + "n01616318", + "n01630670", + "n01632777", + "n01644373", + "n01677366", + "n01694178", + "n01748264", + "n01770393", + "n01774750", + "n01784675", + "n01806143", + "n01820546", + "n01833805", + "n01843383", + "n01847000", + "n01855672", + "n01860187", + "n01882714", + "n01910747", + "n01944390", + "n01983481", + "n01986214", + "n02007558", + "n02009912", + "n02051845", + "n02056570", + "n02066245", + "n02071294", + "n02077923", + "n02085620", + "n02086240", + "n02088094", + "n02088238", + "n02088364", + "n02088466", + "n02091032", + "n02091134", + "n02092339", + "n02094433", + "n02096585", + "n02097298", + "n02098286", + "n02099601", + "n02099712", + "n02102318", + "n02106030", + "n02106166", + "n02106550", + "n02106662", + "n02108089", + "n02108915", + "n02109525", + "n02110185", + "n02110341", + "n02110958", + "n02112018", + "n02112137", + "n02113023", + "n02113624", + "n02113799", + "n02114367", + "n02117135", + "n02119022", + "n02123045", + "n02128385", + "n02128757", + "n02129165", + "n02129604", + "n02130308", + "n02134084", + "n02138441", + "n02165456", + "n02190166", + "n02206856", + "n02219486", + "n02226429", + "n02233338", + "n02236044", + "n02268443", + "n02279972", + "n02317335", + "n02325366", + "n02346627", + "n02356798", + "n02363005", + "n02364673", + "n02391049", + "n02395406", + "n02398521", + "n02410509", + "n02423022", + "n02437616", + "n02445715", + "n02447366", + "n02480495", + "n02480855", + "n02481823", + "n02483362", + "n02486410", + "n02510455", + "n02526121", + "n02607072", + "n02655020", + "n02672831", + "n02701002", + "n02749479", + "n02769748", + "n02793495", + "n02797295", + "n02802426", + "n02808440", + "n02814860", + "n02823750", + "n02841315", + "n02843684", + "n02883205", + "n02906734", + "n02909870", + "n02939185", + "n02948072", + "n02950826", + "n02951358", + "n02966193", + "n02980441", + "n02992529", + "n03124170", + "n03272010", + "n03345487", + "n03372029", + "n03424325", + "n03452741", + "n03467068", + "n03481172", + "n03494278", + "n03495258", + "n03498962", + "n03594945", + "n03602883", + "n03630383", + "n03649909", + "n03676483", + "n03710193", + "n03773504", + "n03775071", + "n03888257", + "n03930630", + "n03947888", + "n04086273", + "n04118538", + "n04133789", + "n04141076", + "n04146614", + "n04147183", + "n04192698", + "n04254680", + "n04266014", + "n04275548", + "n04310018", + "n04325704", + "n04347754", + "n04389033", + "n04409515", + "n04465501", + "n04487394", + "n04522168", + "n04536866", + "n04552348", + "n04591713", + "n07614500", + "n07693725", + "n07695742", + "n07697313", + "n07697537", + "n07714571", + "n07714990", + "n07718472", + "n07720875", + "n07734744", + "n07742313", + "n07745940", + "n07749582", + "n07753275", + "n07753592", + "n07768694", + "n07873807", + "n07880968", + "n07920052", + "n09472597", + "n09835506", + "n10565667", + "n12267677", + } + imagenet_r_mask = [ + wnid in imagenet_r_wnids for wnid in all_imagenet_wordnet_ids + ] + ds = ImageFolder(root=root, transform=transform, **kwargs) + ds.classes = default_classnames["imagenet1k"] + ds.classes = [cl for cl, mask in zip(ds.classes, imagenet_r_mask) if mask] + elif dataset_name == "imagenet-o": + assert split == "test", f"Only `test` split available for {dataset_name}" + # downloadable from https://people.eecs.berkeley.edu/~hendrycks/imagenet-o.tar + if not os.path.exists(root): + print("Downloading imagenet-o...") + call( + "wget https://people.eecs.berkeley.edu/~hendrycks/imagenet-o.tar", + shell=True, + ) + # Untar and move to `root` + call("tar xvf imagenet-o.tar", shell=True) + call(f"mv imagenet-o {root}", shell=True) + ds = ImageFolder(root=root, transform=transform, **kwargs) + ds.classes = default_classnames["imagenet1k"] + imagenet_o_wnids = [ + "n01443537", + "n01704323", + "n01770081", + "n01784675", + "n01819313", + "n01820546", + "n01910747", + "n01917289", + "n01968897", + "n02074367", + "n02317335", + "n02319095", + "n02395406", + "n02454379", + "n02606052", + "n02655020", + "n02666196", + "n02672831", + "n02730930", + "n02777292", + "n02783161", + "n02786058", + "n02787622", + "n02791270", + "n02808304", + "n02817516", + "n02841315", + "n02865351", + "n02877765", + "n02892767", + "n02906734", + "n02910353", + "n02916936", + "n02948072", + "n02965783", + "n03000134", + "n03000684", + "n03017168", + "n03026506", + "n03032252", + "n03075370", + "n03109150", + "n03126707", + "n03134739", + "n03160309", + "n03196217", + "n03207743", + "n03218198", + "n03223299", + "n03240683", + "n03271574", + "n03291819", + "n03297495", + "n03314780", + "n03325584", + "n03344393", + "n03347037", + "n03372029", + "n03376595", + "n03388043", + "n03388183", + "n03400231", + "n03445777", + "n03457902", + "n03467068", + "n03482405", + "n03483316", + "n03494278", + "n03530642", + "n03544143", + "n03584829", + "n03590841", + "n03598930", + "n03602883", + "n03649909", + "n03661043", + "n03666591", + "n03676483", + "n03692522", + "n03706229", + "n03717622", + "n03720891", + "n03721384", + "n03724870", + "n03729826", + "n03733131", + "n03733281", + "n03742115", + "n03786901", + "n03788365", + "n03794056", + "n03804744", + "n03814639", + "n03814906", + "n03825788", + "n03840681", + "n03843555", + "n03854065", + "n03857828", + "n03868863", + "n03874293", + "n03884397", + "n03891251", + "n03908714", + "n03920288", + "n03929660", + "n03930313", + "n03937543", + "n03942813", + "n03944341", + "n03961711", + "n03970156", + "n03982430", + "n03991062", + "n03995372", + "n03998194", + "n04005630", + "n04023962", + "n04033901", + "n04040759", + "n04067472", + "n04074963", + "n04116512", + "n04118776", + "n04125021", + "n04127249", + "n04131690", + "n04141975", + "n04153751", + "n04154565", + "n04201297", + "n04204347", + "n04209133", + "n04209239", + "n04228054", + "n04235860", + "n04243546", + "n04252077", + "n04254120", + "n04258138", + "n04265275", + "n04270147", + "n04275548", + "n04330267", + "n04332243", + "n04336792", + "n04347754", + "n04371430", + "n04371774", + "n04372370", + "n04376876", + "n04409515", + "n04417672", + "n04418357", + "n04423845", + "n04429376", + "n04435653", + "n04442312", + "n04482393", + "n04501370", + "n04507155", + "n04525305", + "n04542943", + "n04554684", + "n04557648", + "n04562935", + "n04579432", + "n04591157", + "n04597913", + "n04599235", + "n06785654", + "n06874185", + "n07615774", + "n07693725", + "n07695742", + "n07697537", + "n07711569", + "n07714990", + "n07715103", + "n07716358", + "n07717410", + "n07718472", + "n07720875", + "n07742313", + "n07745940", + "n07747607", + "n07749582", + "n07753275", + "n07753592", + "n07754684", + "n07768694", + "n07836838", + "n07871810", + "n07873807", + "n07880968", + "n09229709", + "n09472597", + "n12144580", + "n12267677", + "n13052670", + ] + imagenet_o_mask = [ + wnid in set(imagenet_o_wnids) for wnid in all_imagenet_wordnet_ids + ] + ds.classes = [cl for cl, mask in zip(ds.classes, imagenet_o_mask) if mask] + elif dataset_name == "objectnet": + assert split == "test", f"Only `test` split available for {dataset_name}" + # downloadable from https://objectnet.dev/downloads/objectnet-1.0.zip or https://www.dropbox.com/s/raw/cxeztdtm16nzvuw/objectnet-1.0.zip + if not os.path.exists(root): + print("Downloading objectnet...") + call("wget https://objectnet.dev/downloads/objectnet-1.0.zip", shell=True) + # Untar and move to `root` + call( + "UNZIP_DISABLE_ZIPBOMB_DETECTION=TRUE unzip -P objectnetisatestset objectnet-1.0.zip", + shell=True, + ) + os.makedirs(root) + call(f"mv objectnet-1.0 {root}", shell=True) + call(f"cp {root}/objectnet-1.0/mappings/* {root}", shell=True) + ds = objectnet.ObjectNetDataset(root=root, transform=transform) + elif dataset_name == "voc2007": + assert split in ( + "train", + "test", + ), f"Only `train` and `test` split available for {dataset_name}" + ds = voc2007.PASCALVoc2007Cropped( + root=root, set=split, transform=transform, download=download, **kwargs + ) + elif dataset_name == "voc2007_multilabel": + assert split in ( + "train", + "test", + ), f"Only `train` and `test` split available for {dataset_name}" + ds = voc2007.PASCALVoc2007( + root=root, set=split, transform=transform, download=download, **kwargs + ) + elif dataset_name == "aro_visual_attribution": + images_dir = os.path.join(root, "images") + annotation_file = os.path.join(root, "annotations.json") + ds = pos_neg_caption_dataset.PosNegCaptionDataset( + root=images_dir, + ann_file=annotation_file, + transform=transform, + crop_images=True, + **kwargs, + ) + elif dataset_name.startswith("sugar_crepe"): + # https://github.com/RAIVNLab/sugar-crepe/tree/main + base_dir_name, task = dataset_name.split("/") + assert task in ( + "add_att", + "add_obj", + "replace_att", + "replace_obj", + "replace_rel", + "swap_att", + "swap_obj", + ), f"Unknown task {task} for {dataset_name}" + assert split == "test", f"Only `test` split available for {dataset_name}" + dataset_dir = os.path.join(os.path.dirname(root.rstrip("/")), base_dir_name) + images_dir = os.path.join(dataset_dir, "val2017") + annotation_file = os.path.join(dataset_dir, f"{task}.json") + ds = pos_neg_caption_dataset.PosNegCaptionDataset( + root=images_dir, ann_file=annotation_file, transform=transform, **kwargs + ) + elif dataset_name == "winoground": + ds = winoground.WinoGround(root=root, transform=transform) + elif dataset_name == "mscoco_captions": + # https://github.com/mehdidc/retrieval_annotations/releases/tag/1.0.0(annotations) + if split == "train": + archive_name = "train2014.zip" + elif split in ("val", "test"): + archive_name = "val2014.zip" + else: + raise ValueError( + f"split should be `train` or `val` or `test` for `{dataset_name}`" + ) + root_split = os.path.join(root, archive_name.replace(".zip", "")) + if not os.path.exists(root_split): + print(f"Downloading mscoco_captions {archive_name}...") + if not os.path.exists(os.path.join(root, archive_name)): + call( + f"wget http://images.cocodataset.org/zips/{archive_name} --output-document={root}/{archive_name}", + shell=True, + ) + call(f"unzip {root}/{archive_name} -d {root}", shell=True) + if not annotation_file: + annotation_file = f"{root}/coco_{split}_karpathy.json" + if not os.path.exists(annotation_file): + call( + f"wget https://github.com/mehdidc/retrieval_annotations/releases/download/1.0.0/coco_{split}_karpathy.json --output-document={annotation_file}", + shell=True, + ) + ds = CocoCaptions( + root=root_split, annFile=annotation_file, transform=transform, **kwargs + ) + elif dataset_name == "multilingual_mscoco_captions": + from clip_benchmark.datasets import multilingual_mscoco + + if language not in multilingual_mscoco.SUPPORTED_LANGUAGES: + raise ValueError("Unsupported language for multilingual_ms_coco:", language) + + annotation_file = os.path.join( + root, multilingual_mscoco.OUTPUT_FILENAME_TEMPLATE.format(language) + ) + if not os.path.exists(annotation_file): + multilingual_mscoco.create_annotation_file(root, language) + + ds = multilingual_mscoco.Multilingual_MSCOCO( + root=root, ann_file=annotation_file, transform=transform, **kwargs + ) + elif dataset_name == "crossmodal3600": + from clip_benchmark.datasets import crossmodal3600 + + if language not in crossmodal3600.SUPPORTED_LANGUAGES: + raise ValueError("Unsupported language for Crossmodal-3600:", language) + + annotation_file = os.path.join( + root, crossmodal3600.OUTPUT_FILENAME_TEMPLATE.format(language) + ) + if not os.path.exists(annotation_file): + crossmodal3600.create_annotation_file(root, language) + + ds = crossmodal3600.Crossmodal3600( + root=root, ann_file=annotation_file, transform=transform, **kwargs + ) + elif dataset_name == "xtd200": + from clip_benchmark.datasets import xtd200 + + if language not in xtd200.SUPPORTED_LANGUAGES: + raise ValueError("Unsupported language for xtd200:", language) + + annotation_file = os.path.join( + root, xtd200.OUTPUT_FILENAME_TEMPLATE.format(language) + ) + if not os.path.exists(annotation_file): + xtd200.create_annotation_file(root, language) + + ds = xtd200.XTD200( + root=root, ann_file=annotation_file, transform=transform, **kwargs + ) + elif dataset_name == "flickr30k-200": + from clip_benchmark.datasets import flickr30k_200 + + if language not in flickr30k_200.SUPPORTED_LANGUAGES: + raise ValueError("Unsupported language for flickr30k-200:", language) + + annotation_file = os.path.join( + root, flickr30k_200.OUTPUT_FILENAME_TEMPLATE.format(language) + ) + if not os.path.exists(annotation_file): + flickr30k_200.create_annotation_file(root, language) + + ds = flickr30k_200.Flickr30k_200( + root=root, ann_file=annotation_file, transform=transform, **kwargs + ) + elif dataset_name == "flickr30k": + # downloadable from https://www.kaggle.com/datasets/adityajn105/flickr30k + # https://github.com/mehdidc/retrieval_annotations/releases/tag/1.0.0(annotations) + # `kaggle datasets download -d adityajn105/flickr30k` + assert split in ( + "train", + "val", + "test", + ), f"Only `train` and `val` and `test` split available for {dataset_name}" + if not os.path.exists(root): + # Automatic download + print("Downloading flickr30k...") + if not has_kaggle(): + print( + "Kaggle is needed to download the dataset. Please install it via `pip install kaggle`" + ) + sys.exit(1) + call( + "kaggle datasets download -d hsankesara/flickr-image-dataset", + shell=True, + ) + call(f"unzip flickr-image-dataset.zip", shell=True) + call( + f"mv flickr30k_images/flickr30k_images {root} && rm -rf flickr30k_images", + shell=True, + ) + if not annotation_file: + if language == "en": + annotation_file = f"{root}/flickr30k_{split}_karpathy.txt" + elif language == "zh": + annotation_file = f"{root}/flickr30k_{split}_zh.txt" + else: + raise ValueError( + f"Unsupported language {language} for `{dataset_name}`" + ) + if not os.path.exists(annotation_file): + # Download Flickr30K Karpathy test set + if language == "en": + call( + f"wget https://github.com/mehdidc/retrieval_annotations/releases/download/1.0.0/flickr30k_{split}_karpathy.txt --output-document={annotation_file}", + shell=True, + ) + elif language == "zh": + call( + f"wget https://github.com/mehdidc/retrieval_annotations/releases/download/1.0.0/flickr30k_{split}_zh.txt --output-document={annotation_file}", + shell=True, + ) + else: + raise ValueError( + f"Unsupported language {language} for `{dataset_name}`" + ) + ds = flickr.Flickr( + root=root, ann_file=annotation_file, transform=transform, **kwargs + ) + elif dataset_name == "flickr8k": + assert split in ( + "train", + "val", + "test", + ), f"Only `train` and `val` and `test` split available for {dataset_name}" + # downloadable from https://www.kaggle.com/datasets/adityajn105/flickr8k + # `kaggle datasets download -d adityajn105/flickr8k` + # https://github.com/mehdidc/retrieval_annotations/releases/tag/1.0.0(annotations) + if not os.path.exists(root): + # Automatic download + print("Downloading flickr8k...") + if not has_kaggle(): + print( + "Kaggle is needed to download the dataset. Please install it via `pip install kaggle`" + ) + sys.exit(1) + call("kaggle datasets download -d adityajn105/flickr8k", shell=True) + call(f"unzip flickr8k.zip", shell=True) + call(f"mv Images {root}", shell=True) + call(f"mv captions.txt {root}", shell=True) + if not annotation_file: + if language == "en": + annotation_file = f"{root}/flickr8k_{split}_karpathy.txt" + elif language == "zh": + annotation_file = f"{root}/flickr8k_{split}_zh.txt" + else: + raise ValueError( + f"Unsupported language {language} for `{dataset_name}`" + ) + if not os.path.exists(annotation_file): + # Download Flickr8K Karpathy test set + if language == "en": + call( + f"wget https://github.com/mehdidc/retrieval_annotations/releases/download/1.0.0/flickr8k_{split}_karpathy.txt --output-document={annotation_file}", + shell=True, + ) + elif language == "zh": + call( + f"wget https://github.com/mehdidc/retrieval_annotations/releases/download/1.0.0/flickr8k_{split}_zh.txt --output-document={annotation_file}", + shell=True, + ) + else: + raise ValueError( + f"Unsupported language {language} for `{dataset_name}`" + ) + ds = flickr.Flickr( + root=root, ann_file=annotation_file, transform=transform, **kwargs + ) + elif dataset_name == "food101": + assert split in ( + "train", + "test", + ), f"Only `train` and `test` split available for {dataset_name}" + ds = Food101( + root=root, split=split, transform=transform, download=download, **kwargs + ) + # we use the default class names, we just replace "_" by spaces + # to delimit words + ds.classes = [cl.replace("_", " ") for cl in ds.classes] + elif dataset_name == "sun397": + warnings.warn( + f"split argument ignored for `{dataset_name}`, there are no pre-defined train/test splits for this dataset" + ) + # we use the default class names, we just replace "_" and "/" by spaces + # to delimit words + ds = SUN397(root=root, transform=transform, download=download, **kwargs) + ds.classes = [cl.replace("_", " ").replace("/", " ") for cl in ds.classes] + elif dataset_name == "cars": + assert split in ( + "train", + "test", + ), f"Only `train` and `test` split available for {dataset_name}" + ds = StanfordCars( + root=root, split=split, transform=transform, download=download, **kwargs + ) + elif dataset_name == "fgvc_aircraft": + assert split in ( + "train", + "val", + "trainval", + "test", + ), f"Only `train` and `val` and `trainval` and `test` split available for {dataset_name}" + ds = FGVCAircraft( + root=root, + annotation_level="variant", + split=split, + transform=transform, + download=download, + **kwargs, + ) + elif dataset_name == "dtd": + assert split in ( + "train", + "val", + "test", + ), f"Only `train` and `val` and `test` split available for {dataset_name}" + ds = DTD( + root=root, split=split, transform=transform, download=download, **kwargs + ) + elif dataset_name == "pets": + assert split in ( + "trainval", + "test", + ), f"Only `trainval` and `test` split available for {dataset_name}" + ds = OxfordIIITPet( + root=root, + split=split, + target_types="category", + transform=transform, + download=download, + **kwargs, + ) + elif dataset_name == "caltech101": + warnings.warn( + f"split argument ignored for `{dataset_name}`, there are no pre-defined train/test splits for this dataset" + ) + # broken download link (can't download google drive), fixed by this PR https://github.com/pytorch/vision/pull/5645 + # also available in "vtab/caltech101" using VTAB splits, we advice to use VTAB version rather than this one + # since in this one (torchvision) there are no pre-defined test splits + ds = caltech101.Caltech101( + root=root, + target_type="category", + transform=transform, + download=download, + **kwargs, + ) + ds.classes = default_classnames["caltech101"] + elif dataset_name == "flowers": + assert split in ( + "train", + "val", + "test", + ), f"Only `train` and `val` and `test` split available for {dataset_name}" + ds = Flowers102( + root=root, split=split, transform=transform, download=download, **kwargs + ) + # class indices started by 1 until it was fixed in a PR (#TODO link of the PR) + # if older torchvision version, fix it using a target transform that decrements label index + # TODO figure out minimal torchvision version needed instead of decrementing + if ds[0][1] == 1: + ds.target_transform = lambda y: y - 1 + ds.classes = default_classnames["flowers"] + elif dataset_name == "mnist": + assert split in ( + "train", + "test", + ), f"Only `train` and `test` split available for {dataset_name}" + ds = MNIST( + root=root, train=train, transform=transform, download=download, **kwargs + ) + ds.classes = default_classnames["mnist"] + elif dataset_name == "stl10": + assert split in ( + "train", + "test", + ), f"Only `train` and `test` split available for {dataset_name}" + ds = STL10( + root=root, split=split, transform=transform, download=download, **kwargs + ) + elif dataset_name == "eurosat": + warnings.warn( + f"split argument ignored for `{dataset_name}`, there are no pre-defined train/test splits for this dataset" + ) + ds = EuroSAT(root=root, transform=transform, download=download, **kwargs) + ds.classes = default_classnames["eurosat"] + elif dataset_name == "gtsrb": + assert split in ( + "train", + "test", + ), f"Only `train` and `test` split available for {dataset_name}" + ds = GTSRB( + root=root, split=split, transform=transform, download=download, **kwargs + ) + ds.classes = default_classnames["gtsrb"] + elif dataset_name == "country211": + assert split in ( + "train", + "valid", + "test", + ), f"Only `train` and `valid` and `test` split available for {dataset_name}" + ds = Country211( + root=root, split=split, transform=transform, download=download, **kwargs + ) + ds.classes = default_classnames["country211"] + elif dataset_name == "pcam": + assert split in ( + "train", + "val", + "test", + ), f"Only `train` and `val` and `test` split available for {dataset_name}" + # Dead link. Fixed by this PR on torchvision https://github.com/pytorch/vision/pull/5645 + # TODO figure out minimal torchvision version needed + ds = PCAM( + root=root, split=split, transform=transform, download=download, **kwargs + ) + ds.classes = default_classnames["pcam"] + elif dataset_name == "renderedsst2": + assert split in ( + "train", + "val", + "test", + ), f"Only `train` and `val` and `test` split available for {dataset_name}" + ds = RenderedSST2( + root=root, split=split, transform=transform, download=download, **kwargs + ) + elif dataset_name == "fer2013": + assert split in ( + "train", + "test", + ), f"Only `train` and `test` split available for {dataset_name}" + # Downloadable from https://www.kaggle.com/datasets/msambare/fer2013 + # `kaggle datasets download -d msambare/fer2013` + if not os.path.exists(root): + # Automatic download + print("Downloading fer2013...") + if not has_kaggle(): + print( + "Kaggle is needed to download the dataset. Please install it via `pip install kaggle`" + ) + sys.exit(1) + call("kaggle datasets download -d msambare/fer2013", shell=True) + call(f"unzip fer2013.zip -d {root}", shell=True) + root = os.path.join(root, "train" if train else "test") + ds = ImageFolder(root=root, transform=transform) + ds.classes = default_classnames["fer2013"] + elif dataset_name.startswith("tfds/"): + # TFDS datasets support using `timm` and `tensorflow_datasets` + prefix, *name_list = dataset_name.split("/") + name = "/".join(name_list) + ds = build_tfds_dataset( + name, download=download, split=split, data_dir=root, transform=transform + ) + elif dataset_name.startswith("vtab/"): + # VTAB datasets support using `tensorflow_datasets` and `task_adaptation` + prefix, *name_list = dataset_name.split("/") + name = "/".join(name_list) + ds = build_vtab_dataset( + name, + download=download, + split=split, + data_dir=root, + transform=transform, + classnames=default_classnames, + ) + elif dataset_name.startswith("wds/"): + # WebDataset support using `webdataset` library + name = dataset_name.split("/", 1)[1] + ds = build_wds_dataset( + name, + transform=transform, + split=split, + data_dir=root, + cache_dir=wds_cache_dir, + ) + # WDS specify classnames and templates on its own. + elif dataset_name == "dummy": + ds = Dummy() + else: + raise ValueError(f"Unsupported dataset: {dataset_name}.") + + default_dataset_for_templates = "imagenet1k" + if ( + dataset_name.startswith("tfds/") + or dataset_name.startswith("vtab/") + or dataset_name.startswith("wds/") + ): + prefix, *rest = dataset_name.split("/") + short_name = "/".join(rest) + # if it's a vtab/tfds/wds/ dataset, we look for e.g. vtab/ + # as well as in the custom template file/classname file, + # whichever is found. + keys_to_lookup = [dataset_name, short_name] + else: + keys_to_lookup = [dataset_name] + + if use_classnames_and_templates: + # Specify templates for the dataset (if needed) + if custom_templates: + # We override with custom templates ONLY if they are provided, + # which is the case when `custom_templates` is loaded. + ds.templates = value_from_first_key_found( + custom_templates, keys=keys_to_lookup + [default_dataset_for_templates] + ) + assert ( + ds.templates is not None + ), f"Templates not specified for {dataset_name}" + elif not hasattr(ds, "templates"): + # No templates specified by the dataset itself, + # so we use templates are packaged with CLIP benchmark + # (loaded from _zeroshot_classification_templates.json). + ds.templates = value_from_first_key_found( + default_templates, keys=keys_to_lookup + [default_dataset_for_templates] + ) + assert ( + ds.templates is not None + ), f"Templates not specified for {dataset_name}" + else: + # dataset has templates already (e.g., WDS case), so we keep it as is. + pass + + # We override with custom classnames ONLY if they are provided. + if custom_classnames: + ds.classes = value_from_first_key_found( + custom_classnames, keys=keys_to_lookup + ) + + assert ds.classes is not None, f"Classes not specified for {dataset_name}" + assert ds.templates is not None, f"Templates not specified for {dataset_name}" + return ds + + +def value_from_first_key_found(dic, keys): + for k in keys: + if k in dic: + return dic[k] + + +class Dummy: + + def __init__(self): + self.classes = ["blank image", "noisy image"] + + def __getitem__(self, i): + return torch.zeros(3, 224, 224), 0 + + def __len__(self): + return 1 + + +def get_dataset_default_task(dataset): + dataset = dataset.split("wds_")[-1] + if dataset in ( + "flickr30k", + "flickr8k", + "mscoco_captions", + "multilingual_mscoco_captions", + "flickr30k-200", + "crossmodal3600", + "xtd200", + "flickr8k_ocr", + "rendered_ocr", + "flickr30k_ocr", + "mscoco_ocr", + "text_cap", + "pug_spar", + "msrvtt", + "imago_video", + "msvd", + "didemo", + "anet", + "clotho-v2", + "audiocaps-audio-text", + "audiocaps-video-text", + "audiocaps-audio-video", + ): + return "zeroshot_retrieval" + elif dataset in ("pug_animals"): + return "multiclass_retrieval" + elif ( + dataset.startswith("sugar_crepe") + or dataset == "winoground" + or dataset == "aro_visual_attribution" + or dataset.startswith("pug_animals") + ): + return "image_caption_selection" + else: + return "zeroshot_classification" + + +def is_video_dataset(dataset): + if dataset in ( + "k400_val", + "k600_val", + "k700_val", + "ucf101_val", + "hmdb_test", + "mitv1_val", + "ssv2_mc_val", + "msrvtt", + "imago_video", + "msvd", + "didemo", + "anet", + "audiocaps-video-text", + "audiocaps-audio-video", + ): + return True + else: + return False + +def is_audio_dataset(dataset): + return dataset in ( + "clotho-v2", + "audiocaps-audio-text", + "audiocaps-audio-video", + ) + +def get_dataset_collate_fn(dataset_name): + dataset_name = dataset_name.split("wds_")[-1] + if ( + dataset_name + in ( + "mscoco_captions", + "multilingual_mscoco_captions", + "flickr30k", + "flickr8k", + "flickr30k-200", + "crossmodal3600", + "xtd200", + "winoground", + "rendered_ocr", + "flickr30k_ocr", + "flickr8k_ocr", + "mscoco_ocr", + "aro_visual_attribution", + "text_cap", + "pug_spar", + "msrvtt", + "imago_video", + "msvd", + "didemo", + "anet", + ) + or dataset_name.startswith("sugar_crepe") + or dataset_name.startswith("pug_animals") + ): + return image_captions_collate_fn + else: + return default_collate + + +def has_gdown(): + return call("which gdown", shell=True) == 0 + + +def has_kaggle(): + return call("which kaggle", shell=True) == 0 + + +def build_vtab_dataset( + dataset_name, transform, download=True, split="test", data_dir="root", classnames=[] +): + # Using VTAB splits instead of default TFDS splits + from .tfds import (VTABIterableDataset, disable_gpus_on_tensorflow, + download_tfds_dataset) + + # avoid Tensorflow owning GPUs to not clash with PyTorch + disable_gpus_on_tensorflow() + + # by default we take classes from TFDS (default behavior if `classes` stays None), + # except for the datasets that will override `classes` (e.g., clevr_*) + classes = None + if dataset_name == "caltech101": + from task_adaptation.data.caltech import Caltech101 + + tfds_dataset = Caltech101(data_dir=data_dir) + classes = classnames["caltech101_vtab"] + elif dataset_name == "cars": + from task_adaptation.data.cars import CarsData + + tfds_dataset = CarsData(data_dir=data_dir) + elif dataset_name in ("cifar10", "cifar100"): + from task_adaptation.data.cifar import CifarData + + tfds_dataset = CifarData( + data_dir=data_dir, num_classes=10 if dataset_name == "cifar10" else 100 + ) + elif dataset_name.startswith("clevr_"): + from task_adaptation.data.clevr import CLEVRData + + task = _extract_task(dataset_name) + assert task in ("count_all", "closest_object_distance") + tfds_dataset = CLEVRData(task=task, data_dir=data_dir) + if task == "count_all": + classes = classnames["clevr_count_all"] + elif task == "closest_object_distance": + classes = classnames["clevr_closest_object_distance"] + else: + raise ValueError(f"non supported: {task}") + elif dataset_name == "cub": + from task_adaptation.data.cub import CUB2011Data + + tfds_dataset = CUB2011Data(data_dir=data_dir) + elif dataset_name == "diabetic_retinopathy": + # Needs manual download from Kaggle + # 1) `kaggle competitions download -c diabetic-retinopathy-detection` on $ROOT/downloads/manual + # 2) extract archives on $ROOT/downloads/manual + if not os.path.exists(data_dir): + # Automatic download + print("Downloading diabetic_retinopathy...") + if not has_kaggle(): + print( + "Kaggle is needed to download the dataset. Please install it via `pip install kaggle`" + ) + sys.exit(1) + os.makedirs(os.path.join(data_dir, "downloads", "manual")) + call( + f"kaggle competitions download -c diabetic-retinopathy-detection -p {data_dir}/downloads/manual", + shell=True, + ) + call( + f"cd {data_dir}/downloads/manual;unzip diabetic-retinopathy-detection.zip;cat train.zip*>train.zip;cat test.zip*>test.zip;unzip train.zip; unzip test.zip;unzip sample.zip;unzip trainLabels.csv.zip", + shell=True, + ) + from task_adaptation.data.diabetic_retinopathy import RetinopathyData + + tfds_dataset = RetinopathyData(config="btgraham-300", data_dir=data_dir) + classes = classnames["diabetic_retinopathy"] + elif dataset_name == "dmlab": + from task_adaptation.data.dmlab import DmlabData + + download_tfds_dataset( + "dmlab", data_dir=data_dir + ) # it's not called in the original VTAB code, so we do it explictly + tfds_dataset = DmlabData(data_dir=data_dir) + classes = classnames["dmlab"] + elif dataset_name.startswith("dsprites_"): + from task_adaptation.data.dsprites import DSpritesData + + task = _extract_task(dataset_name) + assert task in ( + "label_shape", + "label_scale", + "label_orientation", + "label_x_position", + "label_y_position", + ) + tfds_dataset = DSpritesData(task, data_dir=data_dir) + classes = tfds_dataset._dataset_builder.info.features[task].names + elif dataset_name == "dtd": + from task_adaptation.data.dtd import DTDData + + tfds_dataset = DTDData(data_dir=data_dir) + elif dataset_name == "eurosat": + from task_adaptation.data.eurosat import EurosatData + + tfds_dataset = EurosatData(subset="rgb", data_key="image", data_dir=data_dir) + classes = classnames["eurosat"] + elif dataset_name == "food101": + from task_adaptation.data.food101 import Food101Data + + tfds_dataset = Food101Data(data_dir=data_dir) + elif dataset_name == "inaturalist": + from task_adaptation.data.inaturalist import INaturalistData + + tfds_dataset = INaturalistData(data_dir=data_dir, year=2017) + elif dataset_name.startswith("kitti_"): + from .kitti import KittiData + + task = _extract_task(dataset_name) + assert task in ( + "count_all", + "count_left", + "count_far", + "count_near", + "closest_object_distance", + "closest_object_x_location", + "count_vehicles", + "closest_vehicle_distance", + ) + tfds_dataset = KittiData(task=task, data_dir=data_dir) + if task == "closest_vehicle_distance": + classes = classnames["kitti_closest_vehicle_distance"] + else: + raise ValueError(f"Unsupported task: {task}") + elif dataset_name == "flowers": + from task_adaptation.data.oxford_flowers102 import OxfordFlowers102Data + + tfds_dataset = OxfordFlowers102Data(data_dir=data_dir) + elif dataset_name == "pets": + from task_adaptation.data.oxford_iiit_pet import OxfordIIITPetData + + tfds_dataset = OxfordIIITPetData(data_dir=data_dir) + classes = classnames["pets"] + elif dataset_name == "pcam": + from task_adaptation.data.patch_camelyon import PatchCamelyonData + + tfds_dataset = PatchCamelyonData(data_dir=data_dir) + classes = classnames["pcam"] + elif dataset_name == "resisc45": + # Needs download from OneDrive: https://1drv.ms/u/s!AmgKYzARBl5ca3HNaHIlzp_IXjs + # The archive needs to to be put at /downloads/manual then extracted + if not os.path.exists(data_dir): + os.makedirs(os.path.join(data_dir, "downloads", "manual")) + call( + f"wget 'https://onedrive.live.com/download?resid=5C5E061130630A68!107&authkey=!AHHNaHIlzp_IXjs' --output-document={data_dir}/downloads/manual/resisc45.rar", + shell=True, + ) + call(f"cd {data_dir}/downloads/manual;unrar x resisc45.rar", shell=True) + from task_adaptation.data.resisc45 import Resisc45Data + + tfds_dataset = Resisc45Data(data_dir=data_dir) + elif dataset_name.startswith("smallnorb_"): + from task_adaptation.data.smallnorb import SmallNORBData + + task = _extract_task(dataset_name) + assert task in ( + "label_category", + "label_elevation", + "label_azimuth", + "label_lighting", + ) + tfds_dataset = SmallNORBData(predicted_attribute=task, data_dir=data_dir) + classes = tfds_dataset._dataset_builder.info.features[task].names + elif dataset_name == "sun397": + from task_adaptation.data.sun397 import Sun397Data + + # FIXME There is a problem in `sun397`, when TFDS tries download it + # there is an image that cannot be decoded. For the time being + # we will use torchvision's SUN397 instead. + tfds_dataset = Sun397Data(config="tfds", data_dir=data_dir) + elif dataset_name == "svhn": + from task_adaptation.data.svhn import SvhnData + + tfds_dataset = SvhnData(data_dir=data_dir) + classes = classnames["svhn"] + else: + raise ValueError(f"Unsupported dataset: {dataset_name}") + ds = VTABIterableDataset( + tfds_dataset, + input_name="image", + label_name="label", + transform=transform, + target_transform=int, + split=split, + classes=classes, + ) + return ds + + +def build_tfds_dataset( + name, transform, download=True, split="test", data_dir="root", classes=None +): + from .tfds import disable_gpus_on_tensorflow + + disable_gpus_on_tensorflow() + import tensorflow_datasets as tfds + import timm + + builder = tfds.builder(name, data_dir=data_dir) + if download: + builder.download_and_prepare() + splits = list(builder.info.splits.keys()) + assert split in splits, (split, splits) + ds = timm.data.create_dataset( + f"tfds/{name}", data_dir, split=split, transform=transform, target_transform=int + ) + ds.classes = builder.info.features["label"].names if classes is None else classes + return ds + + +def build_wds_dataset( + dataset_name, transform, split="test", data_dir="root", cache_dir=None +): + """ + Load a dataset in WebDataset format. Either local paths or HTTP URLs can be specified. + Expected file structure is: + ``` + data_dir/ + train/ + nshards.txt + 0.tar + 1.tar + ... + test/ + nshards.txt + 0.tar + 1.tar + ... + classnames.txt + zeroshot_classification_templates.txt + dataset_type.txt + ``` + Classnames and templates are required for zeroshot classification, while dataset type + (equal to "retrieval") is required for zeroshot retrieval datasets. + + You can use the `clip_benchmark_export_wds` or corresponding API + (`clip_benchmark.webdataset_builder.convert_dataset`) to convert datasets to this format. + + Set `cache_dir` to a path to cache the dataset, otherwise, no caching will occur. + """ + import webdataset as wds + + def read_txt(fname): + if "://" in fname: + stream = os.popen("curl -L -s --fail '%s'" % fname, "r") + value = stream.read() + if stream.close(): + raise FileNotFoundError("Failed to retreive data") + else: + with open(fname, "r") as file: + value = file.read() + return value + + # Special handling for Huggingface datasets + # Git LFS files have a different file path to access the raw data than other files + if data_dir.startswith("https://huggingface.co/datasets"): + # Format: https://huggingface.co/datasets///tree/ + *split_url_head, _, url_path = data_dir.split("/", 7) + url_head = "/".join(split_url_head) + metadata_dir = "/".join([url_head, "raw", url_path]) + tardata_dir = "/".join([url_head, "resolve", url_path]) + else: + metadata_dir = tardata_dir = data_dir + # Get number of shards + nshards_fname = os.path.join(metadata_dir, split, "nshards.txt") + nshards = int( + read_txt(nshards_fname) + ) # Do not catch FileNotFound, nshards.txt should be mandatory + # Get dataset type (classification or retrieval) + type_fname = os.path.join(metadata_dir, "dataset_type.txt") + try: + dataset_type = read_txt(type_fname).strip().lower() + except FileNotFoundError: + # print("WARNING: dataset_type.txt not found, assuming type=classification") + dataset_type = "classification" + # + filepattern = os.path.join(tardata_dir, split, "{0..%d}.tar" % (nshards - 1)) + # Load webdataset (support WEBP, PNG, and JPG for now) + if not cache_dir or not isinstance(cache_dir, str): + cache_dir = None + dataset = wds.WebDataset( + filepattern, cache_dir=cache_dir, nodesplitter=lambda src: src + ).decode( + wds.autodecode.ImageHandler("pil", extensions=["webp", "png", "jpg", "jpeg"]) + ) + # Load based on classification or retrieval task + if dataset_type == "retrieval": + dataset = dataset.to_tuple(["webp", "png", "jpg", "jpeg"], "txt").map_tuple( + transform, str.splitlines + ) + dataset.classes = dataset.templates = None + elif dataset_type == "multiclass-retrieval": + dataset = dataset.to_tuple(["webp", "png", "jpg", "jpeg"], "txt").map_tuple( + transform, str.splitlines + ) + dataset.retrieval_template = json.load( + open(os.path.join(metadata_dir, "retrieval_template.json")) + ) + else: + label_type = ( + "npy" if dataset_type == "multilabel" else "cls" + ) # Special case for multilabel + dataset = dataset.to_tuple( + ["webp", "png", "jpg", "jpeg"], label_type + ).map_tuple(transform, None) + # Get class names if present + classnames_fname = os.path.join(metadata_dir, "classnames.txt") + try: + dataset.classes = [ + line.strip() for line in read_txt(classnames_fname).splitlines() + ] + except FileNotFoundError: + print("WARNING: classnames.txt not found") + dataset.classes = None + # Get zeroshot classification templates if present + templates_fname = os.path.join( + metadata_dir, "zeroshot_classification_templates.txt" + ) + try: + dataset.templates = [ + line.strip() for line in read_txt(templates_fname).splitlines() + ] + except FileNotFoundError: + print("WARNING: zeroshot_classification_templates.txt not found") + dataset.templates = None + + return dataset + + +def _extract_task(dataset_name): + prefix, *task_name_list = dataset_name.split("_") + task = "_".join(task_name_list) + return task + + +def image_captions_collate_fn(batch): + transposed = list(zip(*batch)) + imgs = default_collate(transposed[0]) + texts = transposed[1] + return imgs, texts + + +def get_dataset_collection_from_file(path): + datasets = [] + for line in open(path).readlines(): + line = line.strip() + if line != "" and not line.startswith("#"): + datasets.append(line) + print(f"Found {len(datasets)} datasets in {path}:") + print(datasets) + return datasets + + +dataset_collection = { + "vtab": [ + "vtab/caltech101", + "vtab/cifar100", + "vtab/clevr_count_all", + "vtab/clevr_closest_object_distance", + "vtab/diabetic_retinopathy", + "vtab/dmlab", + "vtab/dsprites_label_orientation", + "vtab/dsprites_label_x_position", + "vtab/dtd", + "vtab/eurosat", + "vtab/kitti_closest_vehicle_distance", + "vtab/flowers", + "vtab/pets", + "vtab/pcam", + "vtab/resisc45", + "vtab/smallnorb_label_azimuth", + "vtab/smallnorb_label_elevation", + "sun397", + "vtab/svhn", + ], + "vtab+": [ + "imagenet1k", + "imagenetv2", + "imagenet_sketch", + "imagenet-a", + "imagenet-r", + "objectnet", + "fer2013", + "voc2007", + "voc2007_multilabel", + "sun397", + "cars", + "fgvc_aircraft", + "mnist", + "stl10", + "gtsrb", + "country211", + "renderedsst2", + "vtab/caltech101", + "vtab/cifar10", + "vtab/cifar100", + "vtab/clevr_count_all", + "vtab/clevr_closest_object_distance", + "vtab/diabetic_retinopathy", + "vtab/dmlab", + "vtab/dsprites_label_orientation", + "vtab/dsprites_label_x_position", + "vtab/dtd", + "vtab/eurosat", + "vtab/kitti_closest_vehicle_distance", + "vtab/flowers", + "vtab/pets", + "vtab/pcam", + "vtab/resisc45", + "vtab/smallnorb_label_azimuth", + "vtab/smallnorb_label_elevation", + "vtab/svhn", + ], + "retrieval": [ + "mscoco_captions", + "flickr8k", + "flickr30k", + ], + "imagenet_robustness": [ + "imagenetv2", + "imagenet_sketch", + "imagenet-a", + "imagenet-r", + "objectnet", + ], + "sugar_crepe": [ + "sugar_crepe/add_att", + "sugar_crepe/add_obj", + "sugar_crepe/replace_att", + "sugar_crepe/replace_obj", + "sugar_crepe/replace_rel", + "sugar_crepe/swap_att", + "sugar_crepe/swap_obj", + ], +} + +video_classification_datasets = { + "k400_val": { + "media": K400_ROOT, + "labels": None, + "media_type": "video", + "templates": None, + }, + "k600_val": { + "media": K600_ROOT, + "labels": None, + "media_type": "video", + "templates": None, + }, + "k700_val": { + "media": K700_ROOT, + "labels": None, + "media_type": "video", + "templates": None, + }, + "ucf101_val": { + "media": UCF_ROOT, + "labels": UCF_PROMPT, + "media_type": "video", + "templates": None, + }, + "hmdb_test": { + "media": HMDB_ROOT, + "labels": HMDB_PROMPT, + "media_type": "video", + "templates": None, + }, + "mitv1_val": { + "media": MITV1_ROOT, + "labels": None, + "media_type": "video", + "templates": None, + }, + "ssv2_mc_val": { + "media": SSV2_ROOT, + "labels": None, + "media_type": "video", + "templates": None, + }, +} + +# use by imagenet robustness datasets +all_imagenet_wordnet_ids = [ + "n01440764", + "n01443537", + "n01484850", + "n01491361", + "n01494475", + "n01496331", + "n01498041", + "n01514668", + "n01514859", + "n01518878", + "n01530575", + "n01531178", + "n01532829", + "n01534433", + "n01537544", + "n01558993", + "n01560419", + "n01580077", + "n01582220", + "n01592084", + "n01601694", + "n01608432", + "n01614925", + "n01616318", + "n01622779", + "n01629819", + "n01630670", + "n01631663", + "n01632458", + "n01632777", + "n01641577", + "n01644373", + "n01644900", + "n01664065", + "n01665541", + "n01667114", + "n01667778", + "n01669191", + "n01675722", + "n01677366", + "n01682714", + "n01685808", + "n01687978", + "n01688243", + "n01689811", + "n01692333", + "n01693334", + "n01694178", + "n01695060", + "n01697457", + "n01698640", + "n01704323", + "n01728572", + "n01728920", + "n01729322", + "n01729977", + "n01734418", + "n01735189", + "n01737021", + "n01739381", + "n01740131", + "n01742172", + "n01744401", + "n01748264", + "n01749939", + "n01751748", + "n01753488", + "n01755581", + "n01756291", + "n01768244", + "n01770081", + "n01770393", + "n01773157", + "n01773549", + "n01773797", + "n01774384", + "n01774750", + "n01775062", + "n01776313", + "n01784675", + "n01795545", + "n01796340", + "n01797886", + "n01798484", + "n01806143", + "n01806567", + "n01807496", + "n01817953", + "n01818515", + "n01819313", + "n01820546", + "n01824575", + "n01828970", + "n01829413", + "n01833805", + "n01843065", + "n01843383", + "n01847000", + "n01855032", + "n01855672", + "n01860187", + "n01871265", + "n01872401", + "n01873310", + "n01877812", + "n01882714", + "n01883070", + "n01910747", + "n01914609", + "n01917289", + "n01924916", + "n01930112", + "n01943899", + "n01944390", + "n01945685", + "n01950731", + "n01955084", + "n01968897", + "n01978287", + "n01978455", + "n01980166", + "n01981276", + "n01983481", + "n01984695", + "n01985128", + "n01986214", + "n01990800", + "n02002556", + "n02002724", + "n02006656", + "n02007558", + "n02009229", + "n02009912", + "n02011460", + "n02012849", + "n02013706", + "n02017213", + "n02018207", + "n02018795", + "n02025239", + "n02027492", + "n02028035", + "n02033041", + "n02037110", + "n02051845", + "n02056570", + "n02058221", + "n02066245", + "n02071294", + "n02074367", + "n02077923", + "n02085620", + "n02085782", + "n02085936", + "n02086079", + "n02086240", + "n02086646", + "n02086910", + "n02087046", + "n02087394", + "n02088094", + "n02088238", + "n02088364", + "n02088466", + "n02088632", + "n02089078", + "n02089867", + "n02089973", + "n02090379", + "n02090622", + "n02090721", + "n02091032", + "n02091134", + "n02091244", + "n02091467", + "n02091635", + "n02091831", + "n02092002", + "n02092339", + "n02093256", + "n02093428", + "n02093647", + "n02093754", + "n02093859", + "n02093991", + "n02094114", + "n02094258", + "n02094433", + "n02095314", + "n02095570", + "n02095889", + "n02096051", + "n02096177", + "n02096294", + "n02096437", + "n02096585", + "n02097047", + "n02097130", + "n02097209", + "n02097298", + "n02097474", + "n02097658", + "n02098105", + "n02098286", + "n02098413", + "n02099267", + "n02099429", + "n02099601", + "n02099712", + "n02099849", + "n02100236", + "n02100583", + "n02100735", + "n02100877", + "n02101006", + "n02101388", + "n02101556", + "n02102040", + "n02102177", + "n02102318", + "n02102480", + "n02102973", + "n02104029", + "n02104365", + "n02105056", + "n02105162", + "n02105251", + "n02105412", + "n02105505", + "n02105641", + "n02105855", + "n02106030", + "n02106166", + "n02106382", + "n02106550", + "n02106662", + "n02107142", + "n02107312", + "n02107574", + "n02107683", + "n02107908", + "n02108000", + "n02108089", + "n02108422", + "n02108551", + "n02108915", + "n02109047", + "n02109525", + "n02109961", + "n02110063", + "n02110185", + "n02110341", + "n02110627", + "n02110806", + "n02110958", + "n02111129", + "n02111277", + "n02111500", + "n02111889", + "n02112018", + "n02112137", + "n02112350", + "n02112706", + "n02113023", + "n02113186", + "n02113624", + "n02113712", + "n02113799", + "n02113978", + "n02114367", + "n02114548", + "n02114712", + "n02114855", + "n02115641", + "n02115913", + "n02116738", + "n02117135", + "n02119022", + "n02119789", + "n02120079", + "n02120505", + "n02123045", + "n02123159", + "n02123394", + "n02123597", + "n02124075", + "n02125311", + "n02127052", + "n02128385", + "n02128757", + "n02128925", + "n02129165", + "n02129604", + "n02130308", + "n02132136", + "n02133161", + "n02134084", + "n02134418", + "n02137549", + "n02138441", + "n02165105", + "n02165456", + "n02167151", + "n02168699", + "n02169497", + "n02172182", + "n02174001", + "n02177972", + "n02190166", + "n02206856", + "n02219486", + "n02226429", + "n02229544", + "n02231487", + "n02233338", + "n02236044", + "n02256656", + "n02259212", + "n02264363", + "n02268443", + "n02268853", + "n02276258", + "n02277742", + "n02279972", + "n02280649", + "n02281406", + "n02281787", + "n02317335", + "n02319095", + "n02321529", + "n02325366", + "n02326432", + "n02328150", + "n02342885", + "n02346627", + "n02356798", + "n02361337", + "n02363005", + "n02364673", + "n02389026", + "n02391049", + "n02395406", + "n02396427", + "n02397096", + "n02398521", + "n02403003", + "n02408429", + "n02410509", + "n02412080", + "n02415577", + "n02417914", + "n02422106", + "n02422699", + "n02423022", + "n02437312", + "n02437616", + "n02441942", + "n02442845", + "n02443114", + "n02443484", + "n02444819", + "n02445715", + "n02447366", + "n02454379", + "n02457408", + "n02480495", + "n02480855", + "n02481823", + "n02483362", + "n02483708", + "n02484975", + "n02486261", + "n02486410", + "n02487347", + "n02488291", + "n02488702", + "n02489166", + "n02490219", + "n02492035", + "n02492660", + "n02493509", + "n02493793", + "n02494079", + "n02497673", + "n02500267", + "n02504013", + "n02504458", + "n02509815", + "n02510455", + "n02514041", + "n02526121", + "n02536864", + "n02606052", + "n02607072", + "n02640242", + "n02641379", + "n02643566", + "n02655020", + "n02666196", + "n02667093", + "n02669723", + "n02672831", + "n02676566", + "n02687172", + "n02690373", + "n02692877", + "n02699494", + "n02701002", + "n02704792", + "n02708093", + "n02727426", + "n02730930", + "n02747177", + "n02749479", + "n02769748", + "n02776631", + "n02777292", + "n02782093", + "n02783161", + "n02786058", + "n02787622", + "n02788148", + "n02790996", + "n02791124", + "n02791270", + "n02793495", + "n02794156", + "n02795169", + "n02797295", + "n02799071", + "n02802426", + "n02804414", + "n02804610", + "n02807133", + "n02808304", + "n02808440", + "n02814533", + "n02814860", + "n02815834", + "n02817516", + "n02823428", + "n02823750", + "n02825657", + "n02834397", + "n02835271", + "n02837789", + "n02840245", + "n02841315", + "n02843684", + "n02859443", + "n02860847", + "n02865351", + "n02869837", + "n02870880", + "n02871525", + "n02877765", + "n02879718", + "n02883205", + "n02892201", + "n02892767", + "n02894605", + "n02895154", + "n02906734", + "n02909870", + "n02910353", + "n02916936", + "n02917067", + "n02927161", + "n02930766", + "n02939185", + "n02948072", + "n02950826", + "n02951358", + "n02951585", + "n02963159", + "n02965783", + "n02966193", + "n02966687", + "n02971356", + "n02974003", + "n02977058", + "n02978881", + "n02979186", + "n02980441", + "n02981792", + "n02988304", + "n02992211", + "n02992529", + "n02999410", + "n03000134", + "n03000247", + "n03000684", + "n03014705", + "n03016953", + "n03017168", + "n03018349", + "n03026506", + "n03028079", + "n03032252", + "n03041632", + "n03042490", + "n03045698", + "n03047690", + "n03062245", + "n03063599", + "n03063689", + "n03065424", + "n03075370", + "n03085013", + "n03089624", + "n03095699", + "n03100240", + "n03109150", + "n03110669", + "n03124043", + "n03124170", + "n03125729", + "n03126707", + "n03127747", + "n03127925", + "n03131574", + "n03133878", + "n03134739", + "n03141823", + "n03146219", + "n03160309", + "n03179701", + "n03180011", + "n03187595", + "n03188531", + "n03196217", + "n03197337", + "n03201208", + "n03207743", + "n03207941", + "n03208938", + "n03216828", + "n03218198", + "n03220513", + "n03223299", + "n03240683", + "n03249569", + "n03250847", + "n03255030", + "n03259280", + "n03271574", + "n03272010", + "n03272562", + "n03290653", + "n03291819", + "n03297495", + "n03314780", + "n03325584", + "n03337140", + "n03344393", + "n03345487", + "n03347037", + "n03355925", + "n03372029", + "n03376595", + "n03379051", + "n03384352", + "n03388043", + "n03388183", + "n03388549", + "n03393912", + "n03394916", + "n03400231", + "n03404251", + "n03417042", + "n03424325", + "n03425413", + "n03443371", + "n03444034", + "n03445777", + "n03445924", + "n03447447", + "n03447721", + "n03450230", + "n03452741", + "n03457902", + "n03459775", + "n03461385", + "n03467068", + "n03476684", + "n03476991", + "n03478589", + "n03481172", + "n03482405", + "n03483316", + "n03485407", + "n03485794", + "n03492542", + "n03494278", + "n03495258", + "n03496892", + "n03498962", + "n03527444", + "n03529860", + "n03530642", + "n03532672", + "n03534580", + "n03535780", + "n03538406", + "n03544143", + "n03584254", + "n03584829", + "n03590841", + "n03594734", + "n03594945", + "n03595614", + "n03598930", + "n03599486", + "n03602883", + "n03617480", + "n03623198", + "n03627232", + "n03630383", + "n03633091", + "n03637318", + "n03642806", + "n03649909", + "n03657121", + "n03658185", + "n03661043", + "n03662601", + "n03666591", + "n03670208", + "n03673027", + "n03676483", + "n03680355", + "n03690938", + "n03691459", + "n03692522", + "n03697007", + "n03706229", + "n03709823", + "n03710193", + "n03710637", + "n03710721", + "n03717622", + "n03720891", + "n03721384", + "n03724870", + "n03729826", + "n03733131", + "n03733281", + "n03733805", + "n03742115", + "n03743016", + "n03759954", + "n03761084", + "n03763968", + "n03764736", + "n03769881", + "n03770439", + "n03770679", + "n03773504", + "n03775071", + "n03775546", + "n03776460", + "n03777568", + "n03777754", + "n03781244", + "n03782006", + "n03785016", + "n03786901", + "n03787032", + "n03788195", + "n03788365", + "n03791053", + "n03792782", + "n03792972", + "n03793489", + "n03794056", + "n03796401", + "n03803284", + "n03804744", + "n03814639", + "n03814906", + "n03825788", + "n03832673", + "n03837869", + "n03838899", + "n03840681", + "n03841143", + "n03843555", + "n03854065", + "n03857828", + "n03866082", + "n03868242", + "n03868863", + "n03871628", + "n03873416", + "n03874293", + "n03874599", + "n03876231", + "n03877472", + "n03877845", + "n03884397", + "n03887697", + "n03888257", + "n03888605", + "n03891251", + "n03891332", + "n03895866", + "n03899768", + "n03902125", + "n03903868", + "n03908618", + "n03908714", + "n03916031", + "n03920288", + "n03924679", + "n03929660", + "n03929855", + "n03930313", + "n03930630", + "n03933933", + "n03935335", + "n03937543", + "n03938244", + "n03942813", + "n03944341", + "n03947888", + "n03950228", + "n03954731", + "n03956157", + "n03958227", + "n03961711", + "n03967562", + "n03970156", + "n03976467", + "n03976657", + "n03977966", + "n03980874", + "n03982430", + "n03983396", + "n03991062", + "n03992509", + "n03995372", + "n03998194", + "n04004767", + "n04005630", + "n04008634", + "n04009552", + "n04019541", + "n04023962", + "n04026417", + "n04033901", + "n04033995", + "n04037443", + "n04039381", + "n04040759", + "n04041544", + "n04044716", + "n04049303", + "n04065272", + "n04067472", + "n04069434", + "n04070727", + "n04074963", + "n04081281", + "n04086273", + "n04090263", + "n04099969", + "n04111531", + "n04116512", + "n04118538", + "n04118776", + "n04120489", + "n04125021", + "n04127249", + "n04131690", + "n04133789", + "n04136333", + "n04141076", + "n04141327", + "n04141975", + "n04146614", + "n04147183", + "n04149813", + "n04152593", + "n04153751", + "n04154565", + "n04162706", + "n04179913", + "n04192698", + "n04200800", + "n04201297", + "n04204238", + "n04204347", + "n04208210", + "n04209133", + "n04209239", + "n04228054", + "n04229816", + "n04235860", + "n04238763", + "n04239074", + "n04243546", + "n04251144", + "n04252077", + "n04252225", + "n04254120", + "n04254680", + "n04254777", + "n04258138", + "n04259630", + "n04263257", + "n04264628", + "n04265275", + "n04266014", + "n04270147", + "n04273569", + "n04275548", + "n04277352", + "n04285008", + "n04286575", + "n04296562", + "n04310018", + "n04311004", + "n04311174", + "n04317175", + "n04325704", + "n04326547", + "n04328186", + "n04330267", + "n04332243", + "n04335435", + "n04336792", + "n04344873", + "n04346328", + "n04347754", + "n04350905", + "n04355338", + "n04355933", + "n04356056", + "n04357314", + "n04366367", + "n04367480", + "n04370456", + "n04371430", + "n04371774", + "n04372370", + "n04376876", + "n04380533", + "n04389033", + "n04392985", + "n04398044", + "n04399382", + "n04404412", + "n04409515", + "n04417672", + "n04418357", + "n04423845", + "n04428191", + "n04429376", + "n04435653", + "n04442312", + "n04443257", + "n04447861", + "n04456115", + "n04458633", + "n04461696", + "n04462240", + "n04465501", + "n04467665", + "n04476259", + "n04479046", + "n04482393", + "n04483307", + "n04485082", + "n04486054", + "n04487081", + "n04487394", + "n04493381", + "n04501370", + "n04505470", + "n04507155", + "n04509417", + "n04515003", + "n04517823", + "n04522168", + "n04523525", + "n04525038", + "n04525305", + "n04532106", + "n04532670", + "n04536866", + "n04540053", + "n04542943", + "n04548280", + "n04548362", + "n04550184", + "n04552348", + "n04553703", + "n04554684", + "n04557648", + "n04560804", + "n04562935", + "n04579145", + "n04579432", + "n04584207", + "n04589890", + "n04590129", + "n04591157", + "n04591713", + "n04592741", + "n04596742", + "n04597913", + "n04599235", + "n04604644", + "n04606251", + "n04612504", + "n04613696", + "n06359193", + "n06596364", + "n06785654", + "n06794110", + "n06874185", + "n07248320", + "n07565083", + "n07579787", + "n07583066", + "n07584110", + "n07590611", + "n07613480", + "n07614500", + "n07615774", + "n07684084", + "n07693725", + "n07695742", + "n07697313", + "n07697537", + "n07711569", + "n07714571", + "n07714990", + "n07715103", + "n07716358", + "n07716906", + "n07717410", + "n07717556", + "n07718472", + "n07718747", + "n07720875", + "n07730033", + "n07734744", + "n07742313", + "n07745940", + "n07747607", + "n07749582", + "n07753113", + "n07753275", + "n07753592", + "n07754684", + "n07760859", + "n07768694", + "n07802026", + "n07831146", + "n07836838", + "n07860988", + "n07871810", + "n07873807", + "n07875152", + "n07880968", + "n07892512", + "n07920052", + "n07930864", + "n07932039", + "n09193705", + "n09229709", + "n09246464", + "n09256479", + "n09288635", + "n09332890", + "n09399592", + "n09421951", + "n09428293", + "n09468604", + "n09472597", + "n09835506", + "n10148035", + "n10565667", + "n11879895", + "n11939491", + "n12057211", + "n12144580", + "n12267677", + "n12620546", + "n12768682", + "n12985857", + "n12998815", + "n13037406", + "n13040303", + "n13044778", + "n13052670", + "n13054560", + "n13133613", + "n15075141", +] diff --git a/perception_models/apps/pe/clip_benchmark/datasets/caltech101.py b/perception_models/apps/pe/clip_benchmark/datasets/caltech101.py new file mode 100644 index 0000000000000000000000000000000000000000..208304ffe894c7a79b73510b12622ae180272c56 --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/datasets/caltech101.py @@ -0,0 +1,267 @@ +""" +Code adapted from https://github.com/pytorch/vision/blob/main/torchvision/datasets/caltech.py +Modification of caltech101 from torchvision where the background class is not removed +Thanks to the authors of torchvision +""" + +import os +import os.path +from glob import glob +from typing import Any, Callable, List, Optional, Tuple, Union + +from PIL import Image +from torchvision.datasets.utils import (download_and_extract_archive, + verify_str_arg) +from torchvision.datasets.vision import VisionDataset + + +class Caltech101(VisionDataset): + """`Caltech 101 `_ Dataset. + + .. warning:: + + This class needs `scipy `_ to load target files from `.mat` format. + + Args: + root (string): Root directory of dataset where directory + ``caltech101`` exists or will be saved to if download is set to True. + target_type (string or list, optional): Type of target to use, ``category`` or + ``annotation``. Can also be a list to output a tuple with all specified + target types. ``category`` represents the target class, and + ``annotation`` is a list of points from a hand-generated outline. + Defaults to ``category``. + transform (callable, optional): A function/transform that takes in an PIL image + and returns a transformed version. E.g, ``transforms.RandomCrop`` + target_transform (callable, optional): A function/transform that takes in the + target and transforms it. + download (bool, optional): If true, downloads the dataset from the internet and + puts it in root directory. If dataset is already downloaded, it is not + downloaded again. + """ + + def __init__( + self, + root: str, + target_type: Union[List[str], str] = "category", + transform: Optional[Callable] = None, + target_transform: Optional[Callable] = None, + download: bool = False, + ) -> None: + super().__init__( + os.path.join(root, "caltech101"), + transform=transform, + target_transform=target_transform, + ) + os.makedirs(self.root, exist_ok=True) + if isinstance(target_type, str): + target_type = [target_type] + self.target_type = [ + verify_str_arg(t, "target_type", ("category", "annotation")) + for t in target_type + ] + + if download: + self.download() + + if not self._check_integrity(): + raise RuntimeError( + "Dataset not found or corrupted. You can use download=True to download it" + ) + + self.categories = sorted( + os.listdir(os.path.join(self.root, "101_ObjectCategories")) + ) + # self.categories.remove("BACKGROUND_Google") # this is not a real class + + # For some reason, the category names in "101_ObjectCategories" and + # "Annotations" do not always match. This is a manual map between the + # two. Defaults to using same name, since most names are fine. + name_map = { + "Faces": "Faces_2", + "Faces_easy": "Faces_3", + "Motorbikes": "Motorbikes_16", + "airplanes": "Airplanes_Side_2", + } + self.annotation_categories = list( + map(lambda x: name_map[x] if x in name_map else x, self.categories) + ) + + self.index: List[int] = [] + self.y = [] + for i, c in enumerate(self.categories): + n = len(glob(os.path.join(self.root, "101_ObjectCategories", c, "*.jpg"))) + self.index.extend(range(1, n + 1)) + self.y.extend(n * [i]) + + def __getitem__(self, index: int) -> Tuple[Any, Any]: + """ + Args: + index (int): Index + + Returns: + tuple: (image, target) where the type of target specified by target_type. + """ + import scipy.io + + img = Image.open( + os.path.join( + self.root, + "101_ObjectCategories", + self.categories[self.y[index]], + f"image_{self.index[index]:04d}.jpg", + ) + ) + + target: Any = [] + for t in self.target_type: + if t == "category": + target.append(self.y[index]) + elif t == "annotation": + data = scipy.io.loadmat( + os.path.join( + self.root, + "Annotations", + self.annotation_categories[self.y[index]], + f"annotation_{self.index[index]:04d}.mat", + ) + ) + target.append(data["obj_contour"]) + target = tuple(target) if len(target) > 1 else target[0] + + if self.transform is not None: + img = self.transform(img) + + if self.target_transform is not None: + target = self.target_transform(target) + + return img, target + + def _check_integrity(self) -> bool: + # can be more robust and check hash of files + return os.path.exists(os.path.join(self.root, "101_ObjectCategories")) + + def __len__(self) -> int: + return len(self.index) + + def download(self) -> None: + if self._check_integrity(): + print("Files already downloaded and verified") + return + + download_and_extract_archive( + "https://drive.google.com/file/d/137RyRjvTBkBiIfeYBNZBtViDHQ6_Ewsp", + self.root, + filename="101_ObjectCategories.tar.gz", + md5="b224c7392d521a49829488ab0f1120d9", + ) + download_and_extract_archive( + "https://drive.google.com/file/d/175kQy3UsZ0wUEHZjqkUDdNVssr7bgh_m", + self.root, + filename="Annotations.tar", + md5="6f83eeb1f24d99cab4eb377263132c91", + ) + + def extra_repr(self) -> str: + return "Target type: {target_type}".format(**self.__dict__) + + +class Caltech256(VisionDataset): + """`Caltech 256 `_ Dataset. + + Args: + root (string): Root directory of dataset where directory + ``caltech256`` exists or will be saved to if download is set to True. + transform (callable, optional): A function/transform that takes in an PIL image + and returns a transformed version. E.g, ``transforms.RandomCrop`` + target_transform (callable, optional): A function/transform that takes in the + target and transforms it. + download (bool, optional): If true, downloads the dataset from the internet and + puts it in root directory. If dataset is already downloaded, it is not + downloaded again. + """ + + def __init__( + self, + root: str, + transform: Optional[Callable] = None, + target_transform: Optional[Callable] = None, + download: bool = False, + ) -> None: + super().__init__( + os.path.join(root, "caltech256"), + transform=transform, + target_transform=target_transform, + ) + os.makedirs(self.root, exist_ok=True) + + if download: + self.download() + + if not self._check_integrity(): + raise RuntimeError( + "Dataset not found or corrupted. You can use download=True to download it" + ) + + self.categories = sorted( + os.listdir(os.path.join(self.root, "256_ObjectCategories")) + ) + self.index: List[int] = [] + self.y = [] + for i, c in enumerate(self.categories): + n = len( + [ + item + for item in os.listdir( + os.path.join(self.root, "256_ObjectCategories", c) + ) + if item.endswith(".jpg") + ] + ) + self.index.extend(range(1, n + 1)) + self.y.extend(n * [i]) + + def __getitem__(self, index: int) -> Tuple[Any, Any]: + """ + Args: + index (int): Index + + Returns: + tuple: (image, target) where target is index of the target class. + """ + img = Image.open( + os.path.join( + self.root, + "256_ObjectCategories", + self.categories[self.y[index]], + f"{self.y[index] + 1:03d}_{self.index[index]:04d}.jpg", + ) + ) + + target = self.y[index] + + if self.transform is not None: + img = self.transform(img) + + if self.target_transform is not None: + target = self.target_transform(target) + + return img, target + + def _check_integrity(self) -> bool: + # can be more robust and check hash of files + return os.path.exists(os.path.join(self.root, "256_ObjectCategories")) + + def __len__(self) -> int: + return len(self.index) + + def download(self) -> None: + if self._check_integrity(): + print("Files already downloaded and verified") + return + + download_and_extract_archive( + "https://drive.google.com/file/d/1r6o0pSROcV1_VwT4oSjA2FBUSCWGuxLK", + self.root, + filename="256_ObjectCategories.tar", + md5="67b4f42ca05d46448c6bb8ecd2220f6d", + ) diff --git a/perception_models/apps/pe/clip_benchmark/datasets/clotho_v2.py b/perception_models/apps/pe/clip_benchmark/datasets/clotho_v2.py new file mode 100644 index 0000000000000000000000000000000000000000..5f14bca343a36077a0550bcfa12aa4df2ed064e9 --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/datasets/clotho_v2.py @@ -0,0 +1,51 @@ +from torch.utils.data import Dataset +from subprocess import check_call +import os +import pandas +import shutil +import torch + +def cache_file(url, outfile): + if not os.path.exists(outfile): + print("Downloading Clotho V2 dataset...") + os.makedirs(os.path.dirname(outfile), exist_ok=True) + check_call(["curl", "--url", url, "--output", outfile + ".tmp"]) + os.rename(outfile + ".tmp", outfile) + + +class ClothoV2(Dataset): + def __init__(self, transform, location=os.path.expanduser("~/.cache/perception_encoder/clotho_v2")): + self.df = self.setup_dataset(location) + self.transform = transform + + def setup_dataset(self, location): + url = "https://zenodo.org/records/4783391/files/clotho_audio_evaluation.7z?download=1" + compressed_file = os.path.join(location, "clotho_audio_evaluation.7z") + cache_file(url, compressed_file) + + extracted_dir = os.path.join(location, "extracted") + if not os.path.exists(extracted_dir): + if os.path.exists(extracted_dir + ".tmp"): + shutil.rmtree(extracted_dir + ".tmp") + assert shutil.which("7z"), "Please install 7zip to extract the Clotho V2 dataset (`conda install -c conda-forge p7zip`)" + check_call(["7z", "x", compressed_file, f"-o{extracted_dir}.tmp" ]) + os.rename(extracted_dir + ".tmp", extracted_dir) + + url = "https://zenodo.org/records/4783391/files/clotho_captions_evaluation.csv?download=1" + metadata_file = os.path.join(location, "clotho_captions_evaluation.csv") + cache_file(url, metadata_file) + df = pandas.read_csv(metadata_file) + df["path"] = df["file_name"].apply(lambda x: os.path.join(extracted_dir, "evaluation", x)) + return df + + def __len__(self): + return len(self.df) + + def collate_fn(self, batch): + audios, captions = zip(*batch) + return audios, captions + + def __getitem__(self, idx): + row = self.df.iloc[idx] + captions = [row[f"caption_{i}"] for i in range(1, 6)] + return row["path"], captions diff --git a/perception_models/apps/pe/clip_benchmark/datasets/crossmodal3600.py b/perception_models/apps/pe/clip_benchmark/datasets/crossmodal3600.py new file mode 100644 index 0000000000000000000000000000000000000000..582d428d2e89304c0582ed834469e24335293f69 --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/datasets/crossmodal3600.py @@ -0,0 +1,152 @@ +import codecs +import json +import os +from subprocess import call + +from PIL import Image +from torchvision.datasets import VisionDataset + +SUPPORTED_LANGUAGES = [ + "ar", + "bn", + "cs", + "da", + "de", + "el", + "en", + "es", + "fa", + "fi", + "fil", + "fr", + "he", + "hi", + "hr", + "hu", + "id", + "it", + "ja", + "ko", + "mi", + "nl", + "no", + "pl", + "pt", + "quz", + "ro", + "ru", + "sv", + "sw", + "te", + "th", + "tr", + "uk", + "vi", + "zh", +] + +CAPTIONS_DOWNLOAD_URL = "https://google.github.io/crossmodal-3600/web-data/captions.zip" +IMAGES_DOWNLOAD_URL = ( + "https://open-images-dataset.s3.amazonaws.com/crossmodal-3600/images.tgz" +) +OUTPUT_FILENAME_TEMPLATE = "crossmodal3600_captions-{}.json" + + +class Crossmodal3600(VisionDataset): + def __init__(self, root, ann_file, transform=None, target_transform=None): + super().__init__(root, transform=transform, target_transform=target_transform) + self.ann_file = os.path.expanduser(ann_file) + with codecs.open(ann_file, "r", encoding="utf-8") as fp: + data = json.load(fp) + self.data = [ + (img_path, txt) + for img_path, txt in zip(data["image_paths"], data["annotations"]) + ] + + def __getitem__(self, index): + img, captions = self.data[index] + + # Image + img = Image.open(img).convert("RGB") + if self.transform is not None: + img = self.transform(img) + + # Captions + target = [ + captions, + ] + if self.target_transform is not None: + target = self.target_transform(target) + + return img, target + + def __len__(self) -> int: + return len(self.data) + + +def _download_captions(out_path): + os.makedirs(out_path, exist_ok=True) + print("Downloading captions") + call(f"wget {CAPTIONS_DOWNLOAD_URL} -O captions.zip", shell=True) + call(f"unzip captions.zip -d {out_path}", shell=True) + call("rm captions.zip", shell=True) + + +def _download_images(out_path): + os.makedirs(out_path, exist_ok=True) + print("Downloading images") + call(f"wget {IMAGES_DOWNLOAD_URL} -O images.tgz", shell=True) + call(f"tar -xzf images.tgz -C {out_path}", shell=True) + call("rm images.tgz", shell=True) + + +def create_annotation_file(root, lang_code): + if lang_code not in SUPPORTED_LANGUAGES: + raise ValueError( + f"Language code {lang_code} not supported. Supported languages are {SUPPORTED_LANGUAGES}" + ) + data_dir = os.path.join(root, "xm3600") + images_dir = os.path.join(data_dir, "images") + if not os.path.exists(images_dir): + _download_images(images_dir) + captions_path = os.path.join(data_dir, "captions.jsonl") + if not os.path.exists(captions_path): + _download_captions(data_dir) + with open(captions_path, "r", encoding="utf-8") as f: + data = f.readlines() + data = [json.loads(line) for line in data] + + number_of_missing_images = 0 + valid_images, valid_annotations, valid_indicies = [], [], [] + for i, data_item in enumerate(data): + image_id = data_item["image/key"] + image_name = f"{image_id}.jpg" + image_path = os.path.join(images_dir, image_name) + if not os.path.exists(image_path): + print("Missing image file", image_name) + number_of_missing_images += 1 + continue + captions = data_item[lang_code]["caption"] + txt = captions[0] + + valid_images.append(image_path) + valid_annotations.append(txt) + valid_indicies.append(i) + + if number_of_missing_images > 0: + print(f"*** WARNING *** missing {number_of_missing_images} files.") + + with codecs.open( + os.path.join(root, OUTPUT_FILENAME_TEMPLATE.format(lang_code)), + "w", + encoding="utf-8", + ) as fp: + json.dump( + { + "image_paths": valid_images, + "annotations": valid_annotations, + "indicies": valid_indicies, + }, + fp, + ensure_ascii=False, + ) diff --git a/perception_models/apps/pe/clip_benchmark/datasets/en_classnames.json b/perception_models/apps/pe/clip_benchmark/datasets/en_classnames.json new file mode 100644 index 0000000000000000000000000000000000000000..86de297547d0ec07b9b7ffe6345b5af4e433fc2e --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/datasets/en_classnames.json @@ -0,0 +1,1701 @@ +{ + "flowers": [ + "pink primrose", + "hard-leaved pocket orchid", + "canterbury bells", + "sweet pea", + "english marigold", + "tiger lily", + "moon orchid", + "bird of paradise", + "monkshood", + "globe thistle", + "snapdragon", + "colt's foot", + "king protea", + "spear thistle", + "yellow iris", + "globe flower", + "purple coneflower", + "peruvian lily", + "balloon flower", + "giant white arum lily", + "fire lily", + "pincushion flower", + "fritillary", + "red ginger", + "grape hyacinth", + "corn poppy", + "prince of wales feathers", + "stemless gentian", + "artichoke", + "sweet william", + "carnation", + "garden phlox", + "love in the mist", + "mexican aster", + "alpine sea holly", + "ruby-lipped cattleya", + "cape flower", + "great masterwort", + "siam tulip", + "lenten rose", + "barbeton daisy", + "daffodil", + "sword lily", + "poinsettia", + "bolero deep blue", + "wallflower", + "marigold", + "buttercup", + "oxeye daisy", + "common dandelion", + "petunia", + "wild pansy", + "primula", + "sunflower", + "pelargonium", + "bishop of llandaff", + "gaura", + "geranium", + "orange dahlia", + "pink and yellow dahlia", + "cautleya spicata", + "japanese anemone", + "black-eyed susan", + "silverbush", + "californian poppy", + "osteospermum", + "spring crocus", + "bearded iris", + "windflower", + "tree poppy", + "gazania", + "azalea", + "water lily", + "rose", + "thorn apple", + "morning glory", + "passion flower", + "lotus", + "toad lily", + "anthurium", + "frangipani", + "clematis", + "hibiscus", + "columbine", + "desert-rose", + "tree mallow", + "magnolia", + "cyclamen", + "watercress", + "canna lily", + "hippeastrum", + "bee balm", + "air plant", + "foxglove", + "bougainvillea", + "camellia", + "mallow", + "mexican petunia", + "bromelia", + "blanket flower", + "trumpet creeper", + "blackberry lily" + ], + "gtsrb": [ + "red and white circle 20 kph speed limit", + "red and white circle 30 kph speed limit", + "red and white circle 50 kph speed limit", + "red and white circle 60 kph speed limit", + "red and white circle 70 kph speed limit", + "red and white circle 80 kph speed limit", + "end / de-restriction of 80 kph speed limit", + "red and white circle 100 kph speed limit", + "red and white circle 120 kph speed limit", + "red and white circle red car and black car no passing", + "red and white circle red truck and black car no passing", + "red and white triangle road intersection warning", + "white and yellow diamond priority road", + "red and white upside down triangle yield right-of-way", + "stop", + "empty red and white circle", + "red and white circle no truck entry", + "red circle with white horizonal stripe no entry", + "red and white triangle with exclamation mark warning", + "red and white triangle with black left curve approaching warning", + "red and white triangle with black right curve approaching warning", + "red and white triangle with black double curve approaching warning", + "red and white triangle rough / bumpy road warning", + "red and white triangle car skidding / slipping warning", + "red and white triangle with merging / narrow lanes warning", + "red and white triangle with person digging / construction / road work warning", + "red and white triangle with traffic light approaching warning", + "red and white triangle with person walking warning", + "red and white triangle with child and person walking warning", + "red and white triangle with bicyle warning", + "red and white triangle with snowflake / ice warning", + "red and white triangle with deer warning", + "white circle with gray strike bar no speed limit", + "blue circle with white right turn arrow mandatory", + "blue circle with white left turn arrow mandatory", + "blue circle with white forward arrow mandatory", + "blue circle with white forward or right turn arrow mandatory", + "blue circle with white forward or left turn arrow mandatory", + "blue circle with white keep right arrow mandatory", + "blue circle with white keep left arrow mandatory", + "blue circle with white arrows indicating a traffic circle", + "white circle with gray strike bar indicating no passing for cars has ended", + "white circle with gray strike bar indicating no passing for trucks has ended" + ], + "country211": [ + "Andorra", + "United Arab Emirates", + "Afghanistan", + "Antigua and Barbuda", + "Anguilla", + "Albania", + "Armenia", + "Angola", + "Antarctica", + "Argentina", + "Austria", + "Australia", + "Aruba", + "Aland Islands", + "Azerbaijan", + "Bosnia and Herzegovina", + "Barbados", + "Bangladesh", + "Belgium", + "Burkina Faso", + "Bulgaria", + "Bahrain", + "Benin", + "Bermuda", + "Brunei Darussalam", + "Bolivia", + "Bonaire, Saint Eustatius and Saba", + "Brazil", + "Bahamas", + "Bhutan", + "Botswana", + "Belarus", + "Belize", + "Canada", + "DR Congo", + "Central African Republic", + "Switzerland", + "Cote d'Ivoire", + "Cook Islands", + "Chile", + "Cameroon", + "China", + "Colombia", + "Costa Rica", + "Cuba", + "Cabo Verde", + "Curacao", + "Cyprus", + "Czech Republic", + "Germany", + "Denmark", + "Dominica", + "Dominican Republic", + "Algeria", + "Ecuador", + "Estonia", + "Egypt", + "Spain", + "Ethiopia", + "Finland", + "Fiji", + "Falkland Islands", + "Faeroe Islands", + "France", + "Gabon", + "United Kingdom", + "Grenada", + "Georgia", + "French Guiana", + "Guernsey", + "Ghana", + "Gibraltar", + "Greenland", + "Gambia", + "Guadeloupe", + "Greece", + "South Georgia and South Sandwich Is.", + "Guatemala", + "Guam", + "Guyana", + "Hong Kong", + "Honduras", + "Croatia", + "Haiti", + "Hungary", + "Indonesia", + "Ireland", + "Israel", + "Isle of Man", + "India", + "Iraq", + "Iran", + "Iceland", + "Italy", + "Jersey", + "Jamaica", + "Jordan", + "Japan", + "Kenya", + "Kyrgyz Republic", + "Cambodia", + "St. Kitts and Nevis", + "North Korea", + "South Korea", + "Kuwait", + "Cayman Islands", + "Kazakhstan", + "Laos", + "Lebanon", + "St. Lucia", + "Liechtenstein", + "Sri Lanka", + "Liberia", + "Lithuania", + "Luxembourg", + "Latvia", + "Libya", + "Morocco", + "Monaco", + "Moldova", + "Montenegro", + "Saint-Martin", + "Madagascar", + "Macedonia", + "Mali", + "Myanmar", + "Mongolia", + "Macau", + "Martinique", + "Mauritania", + "Malta", + "Mauritius", + "Maldives", + "Malawi", + "Mexico", + "Malaysia", + "Mozambique", + "Namibia", + "New Caledonia", + "Nigeria", + "Nicaragua", + "Netherlands", + "Norway", + "Nepal", + "New Zealand", + "Oman", + "Panama", + "Peru", + "French Polynesia", + "Papua New Guinea", + "Philippines", + "Pakistan", + "Poland", + "Puerto Rico", + "Palestine", + "Portugal", + "Palau", + "Paraguay", + "Qatar", + "Reunion", + "Romania", + "Serbia", + "Russia", + "Rwanda", + "Saudi Arabia", + "Solomon Islands", + "Seychelles", + "Sudan", + "Sweden", + "Singapore", + "St. Helena", + "Slovenia", + "Svalbard and Jan Mayen Islands", + "Slovakia", + "Sierra Leone", + "San Marino", + "Senegal", + "Somalia", + "South Sudan", + "El Salvador", + "Sint Maarten", + "Syria", + "Eswatini", + "Togo", + "Thailand", + "Tajikistan", + "Timor-Leste", + "Turkmenistan", + "Tunisia", + "Tonga", + "Turkey", + "Trinidad and Tobago", + "Taiwan", + "Tanzania", + "Ukraine", + "Uganda", + "United States", + "Uruguay", + "Uzbekistan", + "Vatican", + "Venezuela", + "British Virgin Islands", + "United States Virgin Islands", + "Vietnam", + "Vanuatu", + "Samoa", + "Kosovo", + "Yemen", + "South Africa", + "Zambia", + "Zimbabwe" + ], + "eurosat": [ + "annual crop land", + "forest", + "brushland or shrubland", + "highway or road", + "industrial buildings or commercial buildings", + "pasture land", + "permanent crop land", + "residential buildings or homes or apartments", + "river", + "lake or sea" + ], + "fer2013": [ + "angry", + "disgusted", + "fearful", + "happy", + "neutral", + "sad", + "surprised" + ], + "caltech101": [ + "background", + "off-center face", + "centered face", + "leopard", + "motorbike", + "accordion", + "airplane", + "anchor", + "ant", + "barrel", + "bass", + "beaver", + "binocular", + "bonsai", + "brain", + "brontosaurus", + "buddha", + "butterfly", + "camera", + "cannon", + "side of a car", + "ceiling fan", + "cellphone", + "chair", + "chandelier", + "body of a cougar cat", + "face of a cougar cat", + "crab", + "crayfish", + "crocodile", + "head of a crocodile", + "cup", + "dalmatian", + "dollar bill", + "dolphin", + "dragonfly", + "electric guitar", + "elephant", + "emu", + "euphonium", + "ewer", + "ferry", + "flamingo", + "head of a flamingo", + "garfield", + "gerenuk", + "gramophone", + "grand piano", + "hawksbill", + "headphone", + "hedgehog", + "helicopter", + "ibis", + "inline skate", + "joshua tree", + "kangaroo", + "ketch", + "lamp", + "laptop", + "llama", + "lobster", + "lotus", + "mandolin", + "mayfly", + "menorah", + "metronome", + "minaret", + "nautilus", + "octopus", + "okapi", + "pagoda", + "panda", + "pigeon", + "pizza", + "platypus", + "pyramid", + "revolver", + "rhino", + "rooster", + "saxophone", + "schooner", + "scissors", + "scorpion", + "sea horse", + "snoopy (cartoon beagle)", + "soccer ball", + "stapler", + "starfish", + "stegosaurus", + "stop sign", + "strawberry", + "sunflower", + "tick", + "trilobite", + "umbrella", + "watch", + "water lilly", + "wheelchair", + "wild cat", + "windsor chair", + "wrench", + "yin and yang symbol" + ], + "caltech101_vtab": [ + "accordion", + "airplane", + "anchor", + "ant", + "background", + "barrel", + "bass", + "beaver", + "binocular", + "bonsai", + "brain", + "brontosaurus", + "buddha", + "butterfly", + "camera", + "cannon", + "side of a car", + "ceiling fan", + "cellphone", + "chair", + "chandelier", + "body of a cougar cat", + "face of a cougar cat", + "crab", + "crayfish", + "crocodile", + "head of a crocodile", + "cup", + "dalmatian", + "dollar bill", + "dolphin", + "dragonfly", + "electric guitar", + "elephant", + "emu", + "euphonium", + "ewer", + "off-center face", + "centered face", + "ferry", + "flamingo", + "head of a flamingo", + "garfield", + "gerenuk", + "gramophone", + "grand piano", + "hawksbill", + "headphone", + "hedgehog", + "helicopter", + "ibis", + "inline skate", + "joshua tree", + "kangaroo", + "ketch", + "lamp", + "laptop", + "leopard", + "llama", + "lobster", + "lotus", + "mandolin", + "mayfly", + "menorah", + "metronome", + "minaret", + "motorbike", + "nautilus", + "octopus", + "okapi", + "pagoda", + "panda", + "pigeon", + "pizza", + "platypus", + "pyramid", + "revolver", + "rhino", + "rooster", + "saxophone", + "schooner", + "scissors", + "scorpion", + "sea horse", + "snoopy (cartoon beagle)", + "soccer ball", + "stapler", + "starfish", + "stegosaurus", + "stop sign", + "strawberry", + "sunflower", + "tick", + "trilobite", + "umbrella", + "watch", + "water lilly", + "wheelchair", + "wild cat", + "windsor chair", + "wrench", + "yin and yang symbol" + ], + "imagenet1k": [ + "tench", + "goldfish", + "great white shark", + "tiger shark", + "hammerhead shark", + "electric ray", + "stingray", + "rooster", + "hen", + "ostrich", + "brambling", + "goldfinch", + "house finch", + "junco", + "indigo bunting", + "American robin", + "bulbul", + "jay", + "magpie", + "chickadee", + "American dipper", + "kite (bird of prey)", + "bald eagle", + "vulture", + "great grey owl", + "fire salamander", + "smooth newt", + "newt", + "spotted salamander", + "axolotl", + "American bullfrog", + "tree frog", + "tailed frog", + "loggerhead sea turtle", + "leatherback sea turtle", + "mud turtle", + "terrapin", + "box turtle", + "banded gecko", + "green iguana", + "Carolina anole", + "desert grassland whiptail lizard", + "agama", + "frilled-necked lizard", + "alligator lizard", + "Gila monster", + "European green lizard", + "chameleon", + "Komodo dragon", + "Nile crocodile", + "American alligator", + "triceratops", + "worm snake", + "ring-necked snake", + "eastern hog-nosed snake", + "smooth green snake", + "kingsnake", + "garter snake", + "water snake", + "vine snake", + "night snake", + "boa constrictor", + "African rock python", + "Indian cobra", + "green mamba", + "sea snake", + "Saharan horned viper", + "eastern diamondback rattlesnake", + "sidewinder rattlesnake", + "trilobite", + "harvestman", + "scorpion", + "yellow garden spider", + "barn spider", + "European garden spider", + "southern black widow", + "tarantula", + "wolf spider", + "tick", + "centipede", + "black grouse", + "ptarmigan", + "ruffed grouse", + "prairie grouse", + "peafowl", + "quail", + "partridge", + "african grey parrot", + "macaw", + "sulphur-crested cockatoo", + "lorikeet", + "coucal", + "bee eater", + "hornbill", + "hummingbird", + "jacamar", + "toucan", + "duck", + "red-breasted merganser", + "goose", + "black swan", + "tusker", + "echidna", + "platypus", + "wallaby", + "koala", + "wombat", + "jellyfish", + "sea anemone", + "brain coral", + "flatworm", + "nematode", + "conch", + "snail", + "slug", + "sea slug", + "chiton", + "chambered nautilus", + "Dungeness crab", + "rock crab", + "fiddler crab", + "red king crab", + "American lobster", + "spiny lobster", + "crayfish", + "hermit crab", + "isopod", + "white stork", + "black stork", + "spoonbill", + "flamingo", + "little blue heron", + "great egret", + "bittern bird", + "crane bird", + "limpkin", + "common gallinule", + "American coot", + "bustard", + "ruddy turnstone", + "dunlin", + "common redshank", + "dowitcher", + "oystercatcher", + "pelican", + "king penguin", + "albatross", + "grey whale", + "killer whale", + "dugong", + "sea lion", + "Chihuahua", + "Japanese Chin", + "Maltese", + "Pekingese", + "Shih Tzu", + "King Charles Spaniel", + "Papillon", + "toy terrier", + "Rhodesian Ridgeback", + "Afghan Hound", + "Basset Hound", + "Beagle", + "Bloodhound", + "Bluetick Coonhound", + "Black and Tan Coonhound", + "Treeing Walker Coonhound", + "English foxhound", + "Redbone Coonhound", + "borzoi", + "Irish Wolfhound", + "Italian Greyhound", + "Whippet", + "Ibizan Hound", + "Norwegian Elkhound", + "Otterhound", + "Saluki", + "Scottish Deerhound", + "Weimaraner", + "Staffordshire Bull Terrier", + "American Staffordshire Terrier", + "Bedlington Terrier", + "Border Terrier", + "Kerry Blue Terrier", + "Irish Terrier", + "Norfolk Terrier", + "Norwich Terrier", + "Yorkshire Terrier", + "Wire Fox Terrier", + "Lakeland Terrier", + "Sealyham Terrier", + "Airedale Terrier", + "Cairn Terrier", + "Australian Terrier", + "Dandie Dinmont Terrier", + "Boston Terrier", + "Miniature Schnauzer", + "Giant Schnauzer", + "Standard Schnauzer", + "Scottish Terrier", + "Tibetan Terrier", + "Australian Silky Terrier", + "Soft-coated Wheaten Terrier", + "West Highland White Terrier", + "Lhasa Apso", + "Flat-Coated Retriever", + "Curly-coated Retriever", + "Golden Retriever", + "Labrador Retriever", + "Chesapeake Bay Retriever", + "German Shorthaired Pointer", + "Vizsla", + "English Setter", + "Irish Setter", + "Gordon Setter", + "Brittany dog", + "Clumber Spaniel", + "English Springer Spaniel", + "Welsh Springer Spaniel", + "Cocker Spaniel", + "Sussex Spaniel", + "Irish Water Spaniel", + "Kuvasz", + "Schipperke", + "Groenendael dog", + "Malinois", + "Briard", + "Australian Kelpie", + "Komondor", + "Old English Sheepdog", + "Shetland Sheepdog", + "collie", + "Border Collie", + "Bouvier des Flandres dog", + "Rottweiler", + "German Shepherd Dog", + "Dobermann", + "Miniature Pinscher", + "Greater Swiss Mountain Dog", + "Bernese Mountain Dog", + "Appenzeller Sennenhund", + "Entlebucher Sennenhund", + "Boxer", + "Bullmastiff", + "Tibetan Mastiff", + "French Bulldog", + "Great Dane", + "St. Bernard", + "husky", + "Alaskan Malamute", + "Siberian Husky", + "Dalmatian", + "Affenpinscher", + "Basenji", + "pug", + "Leonberger", + "Newfoundland dog", + "Great Pyrenees dog", + "Samoyed", + "Pomeranian", + "Chow Chow", + "Keeshond", + "brussels griffon", + "Pembroke Welsh Corgi", + "Cardigan Welsh Corgi", + "Toy Poodle", + "Miniature Poodle", + "Standard Poodle", + "Mexican hairless dog (xoloitzcuintli)", + "grey wolf", + "Alaskan tundra wolf", + "red wolf or maned wolf", + "coyote", + "dingo", + "dhole", + "African wild dog", + "hyena", + "red fox", + "kit fox", + "Arctic fox", + "grey fox", + "tabby cat", + "tiger cat", + "Persian cat", + "Siamese cat", + "Egyptian Mau", + "cougar", + "lynx", + "leopard", + "snow leopard", + "jaguar", + "lion", + "tiger", + "cheetah", + "brown bear", + "American black bear", + "polar bear", + "sloth bear", + "mongoose", + "meerkat", + "tiger beetle", + "ladybug", + "ground beetle", + "longhorn beetle", + "leaf beetle", + "dung beetle", + "rhinoceros beetle", + "weevil", + "fly", + "bee", + "ant", + "grasshopper", + "cricket insect", + "stick insect", + "cockroach", + "praying mantis", + "cicada", + "leafhopper", + "lacewing", + "dragonfly", + "damselfly", + "red admiral butterfly", + "ringlet butterfly", + "monarch butterfly", + "small white butterfly", + "sulphur butterfly", + "gossamer-winged butterfly", + "starfish", + "sea urchin", + "sea cucumber", + "cottontail rabbit", + "hare", + "Angora rabbit", + "hamster", + "porcupine", + "fox squirrel", + "marmot", + "beaver", + "guinea pig", + "common sorrel horse", + "zebra", + "pig", + "wild boar", + "warthog", + "hippopotamus", + "ox", + "water buffalo", + "bison", + "ram (adult male sheep)", + "bighorn sheep", + "Alpine ibex", + "hartebeest", + "impala (antelope)", + "gazelle", + "arabian camel", + "llama", + "weasel", + "mink", + "European polecat", + "black-footed ferret", + "otter", + "skunk", + "badger", + "armadillo", + "three-toed sloth", + "orangutan", + "gorilla", + "chimpanzee", + "gibbon", + "siamang", + "guenon", + "patas monkey", + "baboon", + "macaque", + "langur", + "black-and-white colobus", + "proboscis monkey", + "marmoset", + "white-headed capuchin", + "howler monkey", + "titi monkey", + "Geoffroy's spider monkey", + "common squirrel monkey", + "ring-tailed lemur", + "indri", + "Asian elephant", + "African bush elephant", + "red panda", + "giant panda", + "snoek fish", + "eel", + "silver salmon", + "rock beauty fish", + "clownfish", + "sturgeon", + "gar fish", + "lionfish", + "pufferfish", + "abacus", + "abaya", + "academic gown", + "accordion", + "acoustic guitar", + "aircraft carrier", + "airliner", + "airship", + "altar", + "ambulance", + "amphibious vehicle", + "analog clock", + "apiary", + "apron", + "trash can", + "assault rifle", + "backpack", + "bakery", + "balance beam", + "balloon", + "ballpoint pen", + "Band-Aid", + "banjo", + "baluster / handrail", + "barbell", + "barber chair", + "barbershop", + "barn", + "barometer", + "barrel", + "wheelbarrow", + "baseball", + "basketball", + "bassinet", + "bassoon", + "swimming cap", + "bath towel", + "bathtub", + "station wagon", + "lighthouse", + "beaker", + "military hat (bearskin or shako)", + "beer bottle", + "beer glass", + "bell tower", + "baby bib", + "tandem bicycle", + "bikini", + "ring binder", + "binoculars", + "birdhouse", + "boathouse", + "bobsleigh", + "bolo tie", + "poke bonnet", + "bookcase", + "bookstore", + "bottle cap", + "hunting bow", + "bow tie", + "brass memorial plaque", + "bra", + "breakwater", + "breastplate", + "broom", + "bucket", + "buckle", + "bulletproof vest", + "high-speed train", + "butcher shop", + "taxicab", + "cauldron", + "candle", + "cannon", + "canoe", + "can opener", + "cardigan", + "car mirror", + "carousel", + "tool kit", + "cardboard box / carton", + "car wheel", + "automated teller machine", + "cassette", + "cassette player", + "castle", + "catamaran", + "CD player", + "cello", + "mobile phone", + "chain", + "chain-link fence", + "chain mail", + "chainsaw", + "storage chest", + "chiffonier", + "bell or wind chime", + "china cabinet", + "Christmas stocking", + "church", + "movie theater", + "cleaver", + "cliff dwelling", + "cloak", + "clogs", + "cocktail shaker", + "coffee mug", + "coffeemaker", + "spiral or coil", + "combination lock", + "computer keyboard", + "candy store", + "container ship", + "convertible", + "corkscrew", + "cornet", + "cowboy boot", + "cowboy hat", + "cradle", + "construction crane", + "crash helmet", + "crate", + "infant bed", + "Crock Pot", + "croquet ball", + "crutch", + "cuirass", + "dam", + "desk", + "desktop computer", + "rotary dial telephone", + "diaper", + "digital clock", + "digital watch", + "dining table", + "dishcloth", + "dishwasher", + "disc brake", + "dock", + "dog sled", + "dome", + "doormat", + "drilling rig", + "drum", + "drumstick", + "dumbbell", + "Dutch oven", + "electric fan", + "electric guitar", + "electric locomotive", + "entertainment center", + "envelope", + "espresso machine", + "face powder", + "feather boa", + "filing cabinet", + "fireboat", + "fire truck", + "fire screen", + "flagpole", + "flute", + "folding chair", + "football helmet", + "forklift", + "fountain", + "fountain pen", + "four-poster bed", + "freight car", + "French horn", + "frying pan", + "fur coat", + "garbage truck", + "gas mask or respirator", + "gas pump", + "goblet", + "go-kart", + "golf ball", + "golf cart", + "gondola", + "gong", + "gown", + "grand piano", + "greenhouse", + "radiator grille", + "grocery store", + "guillotine", + "hair clip", + "hair spray", + "half-track", + "hammer", + "hamper", + "hair dryer", + "hand-held computer", + "handkerchief", + "hard disk drive", + "harmonica", + "harp", + "combine harvester", + "hatchet", + "holster", + "home theater", + "honeycomb", + "hook", + "hoop skirt", + "gymnastic horizontal bar", + "horse-drawn vehicle", + "hourglass", + "iPod", + "clothes iron", + "carved pumpkin", + "jeans", + "jeep", + "T-shirt", + "jigsaw puzzle", + "rickshaw", + "joystick", + "kimono", + "knee pad", + "knot", + "lab coat", + "ladle", + "lampshade", + "laptop computer", + "lawn mower", + "lens cap", + "letter opener", + "library", + "lifeboat", + "lighter", + "limousine", + "ocean liner", + "lipstick", + "slip-on shoe", + "lotion", + "music speaker", + "loupe magnifying glass", + "sawmill", + "magnetic compass", + "messenger bag", + "mailbox", + "tights", + "one-piece bathing suit", + "manhole cover", + "maraca", + "marimba", + "mask", + "matchstick", + "maypole", + "maze", + "measuring cup", + "medicine cabinet", + "megalith", + "microphone", + "microwave oven", + "military uniform", + "milk can", + "minibus", + "miniskirt", + "minivan", + "missile", + "mitten", + "mixing bowl", + "mobile home", + "ford model t", + "modem", + "monastery", + "monitor", + "moped", + "mortar and pestle", + "graduation cap", + "mosque", + "mosquito net", + "vespa", + "mountain bike", + "tent", + "computer mouse", + "mousetrap", + "moving van", + "muzzle", + "metal nail", + "neck brace", + "necklace", + "baby pacifier", + "notebook computer", + "obelisk", + "oboe", + "ocarina", + "odometer", + "oil filter", + "pipe organ", + "oscilloscope", + "overskirt", + "bullock cart", + "oxygen mask", + "product packet / packaging", + "paddle", + "paddle wheel", + "padlock", + "paintbrush", + "pajamas", + "palace", + "pan flute", + "paper towel", + "parachute", + "parallel bars", + "park bench", + "parking meter", + "railroad car", + "patio", + "payphone", + "pedestal", + "pencil case", + "pencil sharpener", + "perfume", + "Petri dish", + "photocopier", + "plectrum", + "Pickelhaube", + "picket fence", + "pickup truck", + "pier", + "piggy bank", + "pill bottle", + "pillow", + "ping-pong ball", + "pinwheel", + "pirate ship", + "drink pitcher", + "block plane", + "planetarium", + "plastic bag", + "plate rack", + "farm plow", + "plunger", + "Polaroid camera", + "pole", + "police van", + "poncho", + "pool table", + "soda bottle", + "plant pot", + "potter's wheel", + "power drill", + "prayer rug", + "printer", + "prison", + "missile", + "projector", + "hockey puck", + "punching bag", + "purse", + "quill", + "quilt", + "race car", + "racket", + "radiator", + "radio", + "radio telescope", + "rain barrel", + "recreational vehicle", + "fishing casting reel", + "reflex camera", + "refrigerator", + "remote control", + "restaurant", + "revolver", + "rifle", + "rocking chair", + "rotisserie", + "eraser", + "rugby ball", + "ruler measuring stick", + "sneaker", + "safe", + "safety pin", + "salt shaker", + "sandal", + "sarong", + "saxophone", + "scabbard", + "weighing scale", + "school bus", + "schooner", + "scoreboard", + "CRT monitor", + "screw", + "screwdriver", + "seat belt", + "sewing machine", + "shield", + "shoe store", + "shoji screen / room divider", + "shopping basket", + "shopping cart", + "shovel", + "shower cap", + "shower curtain", + "ski", + "balaclava ski mask", + "sleeping bag", + "slide rule", + "sliding door", + "slot machine", + "snorkel", + "snowmobile", + "snowplow", + "soap dispenser", + "soccer ball", + "sock", + "solar thermal collector", + "sombrero", + "soup bowl", + "keyboard space bar", + "space heater", + "space shuttle", + "spatula", + "motorboat", + "spider web", + "spindle", + "sports car", + "spotlight", + "stage", + "steam locomotive", + "through arch bridge", + "steel drum", + "stethoscope", + "scarf", + "stone wall", + "stopwatch", + "stove", + "strainer", + "tram", + "stretcher", + "couch", + "stupa", + "submarine", + "suit", + "sundial", + "sunglasses", + "sunglasses", + "sunscreen", + "suspension bridge", + "mop", + "sweatshirt", + "swim trunks / shorts", + "swing", + "electrical switch", + "syringe", + "table lamp", + "tank", + "tape player", + "teapot", + "teddy bear", + "television", + "tennis ball", + "thatched roof", + "front curtain", + "thimble", + "threshing machine", + "throne", + "tile roof", + "toaster", + "tobacco shop", + "toilet seat", + "torch", + "totem pole", + "tow truck", + "toy store", + "tractor", + "semi-trailer truck", + "tray", + "trench coat", + "tricycle", + "trimaran", + "tripod", + "triumphal arch", + "trolleybus", + "trombone", + "hot tub", + "turnstile", + "typewriter keyboard", + "umbrella", + "unicycle", + "upright piano", + "vacuum cleaner", + "vase", + "vaulted or arched ceiling", + "velvet fabric", + "vending machine", + "vestment", + "viaduct", + "violin", + "volleyball", + "waffle iron", + "wall clock", + "wallet", + "wardrobe", + "military aircraft", + "sink", + "washing machine", + "water bottle", + "water jug", + "water tower", + "whiskey jug", + "whistle", + "hair wig", + "window screen", + "window shade", + "Windsor tie", + "wine bottle", + "airplane wing", + "wok", + "wooden spoon", + "wool", + "split-rail fence", + "shipwreck", + "sailboat", + "yurt", + "website", + "comic book", + "crossword", + "traffic or street sign", + "traffic light", + "dust jacket", + "menu", + "plate", + "guacamole", + "consomme", + "hot pot", + "trifle", + "ice cream", + "popsicle", + "baguette", + "bagel", + "pretzel", + "cheeseburger", + "hot dog", + "mashed potatoes", + "cabbage", + "broccoli", + "cauliflower", + "zucchini", + "spaghetti squash", + "acorn squash", + "butternut squash", + "cucumber", + "artichoke", + "bell pepper", + "cardoon", + "mushroom", + "Granny Smith apple", + "strawberry", + "orange", + "lemon", + "fig", + "pineapple", + "banana", + "jackfruit", + "cherimoya (custard apple)", + "pomegranate", + "hay", + "carbonara", + "chocolate syrup", + "dough", + "meatloaf", + "pizza", + "pot pie", + "burrito", + "red wine", + "espresso", + "tea cup", + "eggnog", + "mountain", + "bubble", + "cliff", + "coral reef", + "geyser", + "lakeshore", + "promontory", + "sandbar", + "beach", + "valley", + "volcano", + "baseball player", + "bridegroom", + "scuba diver", + "rapeseed", + "daisy", + "yellow lady's slipper", + "corn", + "acorn", + "rose hip", + "horse chestnut seed", + "coral fungus", + "agaric", + "gyromitra", + "stinkhorn mushroom", + "earth star fungus", + "hen of the woods mushroom", + "bolete", + "corn cob", + "toilet paper" + ], + "clevr_count_all": [ + "three", + "four", + "five", + "six", + "seven", + "eight", + "nine", + "ten" + ], + "clevr_closest_object_distance": [ + "very nearby", + "nearby", + "near", + "", + "distant", + "very distant" + ], + "mnist": [ + "0", + "1", + "2", + "3", + "4", + "5", + "6", + "7", + "8", + "9" + ], + "svhn": [ + "zero", + "one", + "two", + "three", + "four", + "five", + "six", + "seven", + "eight", + "nine" + ], + "kitti_closest_vehicle_distance": [ + "a photo i took of a car on my left or right side.", + "a photo i took with a car nearby.", + "a photo i took with a car in the distance.", + "a photo i took with no car." + ], + "dmlab": [ + "nearby apple/melon", + "far apple/melon", + "very far apple/melon", + "nearby lemon", + "far lemon", + "very far lemon" + ], + "pets": [ + "Abyssinian", + "American Bulldog", + "American Pit Bull Terrier", + "Basset Hound", + "Beagle", + "Bengal", + "Birman", + "Bombay", + "Boxer", + "British Shorthair", + "Chihuahua", + "Egyptian Mau", + "English Cocker Spaniel", + "English Setter", + "German Shorthaired", + "Great Pyrenees", + "Havanese", + "Japanese Chin", + "Keeshond", + "Leonberger", + "Maine Coon", + "Miniature Pinscher", + "Newfoundland", + "Persian", + "Pomeranian", + "Pug", + "Ragdoll", + "Russian Blue", + "Saint Bernard", + "Samoyed", + "Scottish Terrier", + "Shiba Inu", + "Siamese", + "Sphynx", + "Staffordshire Bull Terrier", + "Wheaten Terrier", + "Yorkshire Terrier" + ], + "pcam": [ + "lymph node", + "lymph node containing metastatic tumor tissue" + ], + "diabetic_retinopathy": [ + "no diabetic retinopathy", + "mild diabetic retinopathy", + "moderate diabetic retinopathy", + "severe diabetic retinopathy", + "proliferative diabetic retinopathy" + ] +} \ No newline at end of file diff --git a/perception_models/apps/pe/clip_benchmark/datasets/en_zeroshot_classification_templates.json b/perception_models/apps/pe/clip_benchmark/datasets/en_zeroshot_classification_templates.json new file mode 100644 index 0000000000000000000000000000000000000000..f2ca0e0f30bd8b05de537bdaba7afa61b0ca0183 --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/datasets/en_zeroshot_classification_templates.json @@ -0,0 +1,485 @@ +{ + "cifar10": [ + "a photo of a {c}.", + "a blurry photo of a {c}.", + "a black and white photo of a {c}.", + "a low contrast photo of a {c}.", + "a high contrast photo of a {c}.", + "a bad photo of a {c}.", + "a good photo of a {c}.", + "a photo of a small {c}.", + "a photo of a big {c}.", + "a photo of the {c}.", + "a blurry photo of the {c}.", + "a black and white photo of the {c}.", + "a low contrast photo of the {c}.", + "a high contrast photo of the {c}.", + "a bad photo of the {c}.", + "a good photo of the {c}.", + "a photo of the small {c}.", + "a photo of the big {c}." + ], + "cifar100": [ + "a photo of a {c}.", + "a blurry photo of a {c}.", + "a black and white photo of a {c}.", + "a low contrast photo of a {c}.", + "a high contrast photo of a {c}.", + "a bad photo of a {c}.", + "a good photo of a {c}.", + "a photo of a small {c}.", + "a photo of a big {c}.", + "a photo of the {c}.", + "a blurry photo of the {c}.", + "a black and white photo of the {c}.", + "a low contrast photo of the {c}.", + "a high contrast photo of the {c}.", + "a bad photo of the {c}.", + "a good photo of the {c}.", + "a photo of the small {c}.", + "a photo of the big {c}." + ], + "imagenet1k": [ + "a bad photo of a {c}.", + "a photo of many {c}.", + "a sculpture of a {c}.", + "a photo of the hard to see {c}.", + "a low resolution photo of the {c}.", + "a rendering of a {c}.", + "graffiti of a {c}.", + "a bad photo of the {c}.", + "a cropped photo of the {c}.", + "a tattoo of a {c}.", + "the embroidered {c}.", + "a photo of a hard to see {c}.", + "a bright photo of a {c}.", + "a photo of a clean {c}.", + "a photo of a dirty {c}.", + "a dark photo of the {c}.", + "a drawing of a {c}.", + "a photo of my {c}.", + "the plastic {c}.", + "a photo of the cool {c}.", + "a close-up photo of a {c}.", + "a black and white photo of the {c}.", + "a painting of the {c}.", + "a painting of a {c}.", + "a pixelated photo of the {c}.", + "a sculpture of the {c}.", + "a bright photo of the {c}.", + "a cropped photo of a {c}.", + "a plastic {c}.", + "a photo of the dirty {c}.", + "a jpeg corrupted photo of a {c}.", + "a blurry photo of the {c}.", + "a photo of the {c}.", + "a good photo of the {c}.", + "a rendering of the {c}.", + "a {c} in a video game.", + "a photo of one {c}.", + "a doodle of a {c}.", + "a close-up photo of the {c}.", + "a photo of a {c}.", + "the origami {c}.", + "the {c} in a video game.", + "a sketch of a {c}.", + "a doodle of the {c}.", + "a origami {c}.", + "a low resolution photo of a {c}.", + "the toy {c}.", + "a rendition of the {c}.", + "a photo of the clean {c}.", + "a photo of a large {c}.", + "a rendition of a {c}.", + "a photo of a nice {c}.", + "a photo of a weird {c}.", + "a blurry photo of a {c}.", + "a cartoon {c}.", + "art of a {c}.", + "a sketch of the {c}.", + "a embroidered {c}.", + "a pixelated photo of a {c}.", + "itap of the {c}.", + "a jpeg corrupted photo of the {c}.", + "a good photo of a {c}.", + "a plushie {c}.", + "a photo of the nice {c}.", + "a photo of the small {c}.", + "a photo of the weird {c}.", + "the cartoon {c}.", + "art of the {c}.", + "a drawing of the {c}.", + "a photo of the large {c}.", + "a black and white photo of a {c}.", + "the plushie {c}.", + "a dark photo of a {c}.", + "itap of a {c}.", + "graffiti of the {c}.", + "a toy {c}.", + "itap of my {c}.", + "a photo of a cool {c}.", + "a photo of a small {c}.", + "a tattoo of the {c}." + ], + "food101": [ + "a photo of {c}, a type of food." + ], + "sun397": [ + "a photo of a {c}.", + "a photo of the {c}." + ], + "cars": [ + "a photo of a {c}.", + "a photo of the {c}.", + "a photo of my {c}.", + "i love my {c}!", + "a photo of my dirty {c}.", + "a photo of my clean {c}.", + "a photo of my new {c}.", + "a photo of my old {c}." + ], + "fgvc_aircraft": [ + "a photo of a {c}, a type of aircraft.", + "a photo of the {c}, a type of aircraft." + ], + "dtd": [ + "a photo of a {c} texture.", + "a photo of a {c} pattern.", + "a photo of a {c} thing.", + "a photo of a {c} object.", + "a photo of the {c} texture.", + "a photo of the {c} pattern.", + "a photo of the {c} thing.", + "a photo of the {c} object." + ], + "pets": [ + "a photo of a {c}, a type of pet." + ], + "caltech101": [ + "a photo of a {c}.", + "a painting of a {c}.", + "a plastic {c}.", + "a sculpture of a {c}.", + "a sketch of a {c}.", + "a tattoo of a {c}.", + "a toy {c}.", + "a rendition of a {c}.", + "a embroidered {c}.", + "a cartoon {c}.", + "a {c} in a video game.", + "a plushie {c}.", + "a origami {c}.", + "art of a {c}.", + "graffiti of a {c}.", + "a drawing of a {c}.", + "a doodle of a {c}.", + "a photo of the {c}.", + "a painting of the {c}.", + "the plastic {c}.", + "a sculpture of the {c}.", + "a sketch of the {c}.", + "a tattoo of the {c}.", + "the toy {c}.", + "a rendition of the {c}.", + "the embroidered {c}.", + "the cartoon {c}.", + "the {c} in a video game.", + "the plushie {c}.", + "the origami {c}.", + "art of the {c}.", + "graffiti of the {c}.", + "a drawing of the {c}.", + "a doodle of the {c}." + ], + "flowers": [ + "a photo of a {c}, a type of flower." + ], + "mnist": [ + "a photo of the number: \"{c}\"." + ], + "stl10": [ + "a photo of a {c}.", + "a photo of the {c}." + ], + "eurosat": [ + "a centered satellite photo of {c}.", + "a centered satellite photo of a {c}.", + "a centered satellite photo of the {c}." + ], + "gtsrb": [ + "a zoomed in photo of a \"{c}\" traffic sign.", + "a centered photo of a \"{c}\" traffic sign.", + "a close up photo of a \"{c}\" traffic sign." + ], + "country211": [ + "a photo i took in {c}.", + "a photo i took while visiting {c}.", + "a photo from my home country of {c}.", + "a photo from my visit to {c}.", + "a photo showing the country of {c}." + ], + "renderedsst2": [ + "a {c} review of a movie." + ], + "voc2007": [ + "a photo of a {c}." + ], + "voc2007_multilabel": [ + "a photo of a {c}." + ], + "fer2013": [ + "a photo of a {c} looking face.", + "a photo of a face showing the emotion: {c}.", + "a photo of a face looking {c}.", + "a face that looks {c}.", + "they look {c}.", + "look at how {c} they are." + ], + "clevr_count_all": [ + "a picture of {c} objects" + ], + "clevr_closest_object_distance": [ + "{c} shapes." + ], + "pcam": [ + "a histopathology slide showing {c}", + "histopathology image of {c}" + ], + "svhn": [ + "a photo of the number {c} written on a sign", + "an outdoor house number {c}", + "the number {c} in the center of the image", + "an outdoor number {c} writte on a sign", + "an outdoor number {c}", + "a centered image of the number {c}" + ], + "resisc45": [ + "a sattelite image of {c}", + "an aerial view of {c}", + "a sattelite photo of {c}", + "{c} from above" + ], + "kitti_closest_vehicle_distance": [ + "{c}" + ], + "smallnorb_label_azimuth": [ + "an object rotated at {c}", + "something rotated at {c}", + "{c} rotation", + "something at a {c} angle" + ], + "smallnorb_label_elevation": [ + "an object rotated at {c}", + "something rotated at {c}", + "{c} rotation", + "something at a {c} angle" + ], + "dsprites_label_x_position": [ + "an object located at position {c}% on the horizontal axis" + ], + "dsprites_label_orientation": [ + "an object rotated at {c}", + "something rotated at {c}", + "{c} rotation", + "something at a {c} angle" + ], + "dmlab": [ + "{c}" + ], + "diabetic_retinopathy": [ + "a retinal image with {c}" + ], + "dummy": [ + "a photo of a {c}" + ], + "k400_val": [ + "a photo of {c}.", + "a photo of a person {c}.", + "a photo of a person using {c}.", + "a photo of a person doing {c}.", + "a photo of a person during {c}.", + "a photo of a person performing {c}.", + "a photo of a person practicing {c}.", + "a video of {c}.", + "a video of a person {c}.", + "a video of a person using {c}.", + "a video of a person doing {c}.", + "a video of a person during {c}.", + "a video of a person performing {c}.", + "a video of a person practicing {c}.", + "a example of {c}.", + "a example of a person {c}.", + "a example of a person using {c}.", + "a example of a person doing {c}.", + "a example of a person during {c}.", + "a example of a person performing {c}.", + "a example of a person practicing {c}.", + "a demonstration of {c}.", + "a demonstration of a person {c}.", + "a demonstration of a person using {c}.", + "a demonstration of a person doing {c}.", + "a demonstration of a person during {c}.", + "a demonstration of a person performing {c}.", + "a demonstration of a person practicing {c}." + ], + "k600_val": [ + "a photo of {c}.", + "a photo of a person {c}.", + "a photo of a person using {c}.", + "a photo of a person doing {c}.", + "a photo of a person during {c}.", + "a photo of a person performing {c}.", + "a photo of a person practicing {c}.", + "a video of {c}.", + "a video of a person {c}.", + "a video of a person using {c}.", + "a video of a person doing {c}.", + "a video of a person during {c}.", + "a video of a person performing {c}.", + "a video of a person practicing {c}.", + "a example of {c}.", + "a example of a person {c}.", + "a example of a person using {c}.", + "a example of a person doing {c}.", + "a example of a person during {c}.", + "a example of a person performing {c}.", + "a example of a person practicing {c}.", + "a demonstration of {c}.", + "a demonstration of a person {c}.", + "a demonstration of a person using {c}.", + "a demonstration of a person doing {c}.", + "a demonstration of a person during {c}.", + "a demonstration of a person performing {c}.", + "a demonstration of a person practicing {c}." + ], + "k700_val": [ + "a photo of {c}.", + "a photo of a person {c}.", + "a photo of a person using {c}.", + "a photo of a person doing {c}.", + "a photo of a person during {c}.", + "a photo of a person performing {c}.", + "a photo of a person practicing {c}.", + "a video of {c}.", + "a video of a person {c}.", + "a video of a person using {c}.", + "a video of a person doing {c}.", + "a video of a person during {c}.", + "a video of a person performing {c}.", + "a video of a person practicing {c}.", + "a example of {c}.", + "a example of a person {c}.", + "a example of a person using {c}.", + "a example of a person doing {c}.", + "a example of a person during {c}.", + "a example of a person performing {c}.", + "a example of a person practicing {c}.", + "a demonstration of {c}.", + "a demonstration of a person {c}.", + "a demonstration of a person using {c}.", + "a demonstration of a person doing {c}.", + "a demonstration of a person during {c}.", + "a demonstration of a person performing {c}.", + "a demonstration of a person practicing {c}." + ], + "ucf101_val": [ + "a photo of a person {c}.", + "a video of a person {c}.", + "a example of a person {c}.", + "a demonstration of a person {c}.", + "a photo of the person {c}.", + "a video of the person {c}.", + "a example of the person {c}.", + "a demonstration of the person {c}.", + "a photo of a person using {c}.", + "a video of a person using {c}.", + "a example of a person using {c}.", + "a demonstration of a person using {c}.", + "a photo of the person using {c}.", + "a video of the person using {c}.", + "a example of the person using {c}.", + "a demonstration of the person using {c}.", + "a photo of a person doing {c}.", + "a video of a person doing {c}.", + "a example of a person doing {c}.", + "a demonstration of a person doing {c}.", + "a photo of the person doing {c}.", + "a video of the person doing {c}.", + "a example of the person doing {c}.", + "a demonstration of the person doing {c}.", + "a photo of a person during {c}.", + "a video of a person during {c}.", + "a example of a person during {c}.", + "a demonstration of a person during {c}.", + "a photo of the person during {c}.", + "a video of the person during {c}.", + "a example of the person during {c}.", + "a demonstration of the person during {c}.", + "a photo of a person performing {c}.", + "a video of a person performing {c}.", + "a example of a person performing {c}.", + "a demonstration of a person performing {c}.", + "a photo of the person performing {c}.", + "a video of the person performing {c}.", + "a example of the person performing {c}.", + "a demonstration of the person performing {c}.", + "a photo of a person practicing {c}.", + "a video of a person practicing {c}.", + "a example of a person practicing {c}.", + "a demonstration of a person practicing {c}.", + "a photo of the person practicing {c}.", + "a video of the person practicing {c}.", + "a example of the person practicing {c}.", + "a demonstration of the person practicing {c}." + ], + "hmdb_test": [ + "a photo of a person {c}.", + "a video of a person {c}.", + "a example of a person {c}.", + "a demonstration of a person {c}.", + "a photo of the person {c}.", + "a video of the person {c}.", + "a example of the person {c}.", + "a demonstration of the person {c}.", + "a photo of a person using {c}.", + "a video of a person using {c}.", + "a example of a person using {c}.", + "a demonstration of a person using {c}.", + "a photo of the person using {c}.", + "a video of the person using {c}.", + "a example of the person using {c}.", + "a demonstration of the person using {c}.", + "a photo of a person doing {c}.", + "a video of a person doing {c}.", + "a example of a person doing {c}.", + "a demonstration of a person doing {c}.", + "a photo of the person doing {c}.", + "a video of the person doing {c}.", + "a example of the person doing {c}.", + "a demonstration of the person doing {c}.", + "a photo of a person during {c}.", + "a video of a person during {c}.", + "a example of a person during {c}.", + "a demonstration of a person during {c}.", + "a photo of the person during {c}.", + "a video of the person during {c}.", + "a example of the person during {c}.", + "a demonstration of the person during {c}.", + "a photo of a person performing {c}.", + "a video of a person performing {c}.", + "a example of a person performing {c}.", + "a demonstration of a person performing {c}.", + "a photo of the person performing {c}.", + "a video of the person performing {c}.", + "a example of the person performing {c}.", + "a demonstration of the person performing {c}.", + "a photo of a person practicing {c}.", + "a video of a person practicing {c}.", + "a example of a person practicing {c}.", + "a demonstration of a person practicing {c}.", + "a photo of the person practicing {c}.", + "a video of the person practicing {c}.", + "a example of the person practicing {c}.", + "a demonstration of the person practicing {c}." + ] +} diff --git a/perception_models/apps/pe/clip_benchmark/datasets/flickr.py b/perception_models/apps/pe/clip_benchmark/datasets/flickr.py new file mode 100644 index 0000000000000000000000000000000000000000..efda2ddcd8e723de5fc2b3eaa2fd57d56b34e7f3 --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/datasets/flickr.py @@ -0,0 +1,62 @@ +""" +Adapted from https://github.com/pytorch/vision/blob/main/torchvision/datasets/flickr.py +Thanks to the authors of torchvision +""" + +import glob +import os +from collections import defaultdict +from html.parser import HTMLParser +from typing import Any, Callable, Dict, List, Optional, Tuple + +from PIL import Image +from torchvision.datasets import VisionDataset + + +class Flickr(VisionDataset): + + def __init__( + self, + root: str, + ann_file: str, + transform: Optional[Callable] = None, + target_transform: Optional[Callable] = None, + ) -> None: + super().__init__(root, transform=transform, target_transform=target_transform) + self.ann_file = os.path.expanduser(ann_file) + data = defaultdict(list) + with open(ann_file) as fd: + fd.readline() + for line in fd: + line = line.strip() + if line: + # some lines have comma in the caption, se we make sure we do the split correctly + img, caption = line.strip().split(".jpg,") + img = img + ".jpg" + data[img].append(caption) + self.data = list(data.items()) + + def __getitem__(self, index: int) -> Tuple[Any, Any]: + """ + Args: + index (int): Index + + Returns: + tuple: Tuple (image, target). target is a list of captions for the image. + """ + img, captions = self.data[index] + + # Image + img = Image.open(os.path.join(self.root, img)).convert("RGB") + if self.transform is not None: + img = self.transform(img) + + # Captions + target = captions + if self.target_transform is not None: + target = self.target_transform(target) + + return img, target + + def __len__(self) -> int: + return len(self.data) diff --git a/perception_models/apps/pe/clip_benchmark/datasets/flickr30k_200.py b/perception_models/apps/pe/clip_benchmark/datasets/flickr30k_200.py new file mode 100644 index 0000000000000000000000000000000000000000..e8c4da53f9422d4c0569d4958daaeed18f06ea3c --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/datasets/flickr30k_200.py @@ -0,0 +1,119 @@ +import codecs +import json +import os +from subprocess import call + +import requests +from PIL import Image +from torchvision.datasets import VisionDataset + +from .flores_langs import flores_languages + +GITHUB_DATA_PATH = ( + "https://raw.githubusercontent.com/visheratin/nllb-clip/main/data/flickr30k-200/" +) +SUPPORTED_LANGUAGES = flores_languages + +IMAGE_INDEX_FILENAME = "filenames.txt" + +CAPTIONS_FILENAME_TEMPLATE = "{}.txt" +OUTPUT_FILENAME_TEMPLATE = "flickr30k_200-{}.json" + +IMAGES_DOWNLOAD_URL = "https://nllb-data.com/test/flickr30k/images.tar.gz" + + +class Flickr30k_200(VisionDataset): + def __init__(self, root, ann_file, transform=None, target_transform=None): + super().__init__(root, transform=transform, target_transform=target_transform) + self.ann_file = os.path.expanduser(ann_file) + with codecs.open(ann_file, "r", encoding="utf-8") as fp: + data = json.load(fp) + self.data = [ + (img_path, txt) + for img_path, txt in zip(data["image_paths"], data["annotations"]) + ] + + def __getitem__(self, index): + img, captions = self.data[index] + + # Image + img = Image.open(img).convert("RGB") + if self.transform is not None: + img = self.transform(img) + + # Captions + target = [ + captions, + ] + if self.target_transform is not None: + target = self.target_transform(target) + + return img, target + + def __len__(self) -> int: + return len(self.data) + + +def _get_lines(url): + response = requests.get(url, timeout=30) + return response.text.splitlines() + + +def _download_images(out_path): + os.makedirs(out_path, exist_ok=True) + print("Downloading images") + call(f"wget {IMAGES_DOWNLOAD_URL} -O images.tar.gz", shell=True) + call(f"tar -xzf images.tar.gz -C {out_path}", shell=True) + call("rm images.tar.gz", shell=True) + + +def create_annotation_file(root, lang_code): + if lang_code not in SUPPORTED_LANGUAGES: + raise ValueError( + f"Language code {lang_code} not supported. Supported languages are {SUPPORTED_LANGUAGES}" + ) + data_dir = os.path.join(root, "flickr30k-200") + if not os.path.exists(data_dir): + _download_images(data_dir) + images_dir = os.path.join(root, "flickr30k-200", "images") + print("Downloading flickr30k-200 index file") + download_path = os.path.join(GITHUB_DATA_PATH, IMAGE_INDEX_FILENAME) + target_images = _get_lines(download_path) + + print("Downloading flickr30k-200 captions:", lang_code) + captions_path = GITHUB_DATA_PATH + download_path = os.path.join( + captions_path, CAPTIONS_FILENAME_TEMPLATE.format(lang_code) + ) + target_captions = _get_lines(download_path) + + number_of_missing_images = 0 + valid_images, valid_annotations, valid_indicies = [], [], [] + for i, (img, txt) in enumerate(zip(target_images, target_captions)): + image_path = os.path.join(images_dir, img) + if not os.path.exists(image_path): + print("Missing image file", img) + number_of_missing_images += 1 + continue + + valid_images.append(image_path) + valid_annotations.append(txt) + valid_indicies.append(i) + + if number_of_missing_images > 0: + print(f"*** WARNING *** missing {number_of_missing_images} files.") + + with codecs.open( + os.path.join(root, OUTPUT_FILENAME_TEMPLATE.format(lang_code)), + "w", + encoding="utf-8", + ) as fp: + json.dump( + { + "image_paths": valid_images, + "annotations": valid_annotations, + "indicies": valid_indicies, + }, + fp, + ensure_ascii=False, + ) diff --git a/perception_models/apps/pe/clip_benchmark/datasets/flores_langs.py b/perception_models/apps/pe/clip_benchmark/datasets/flores_langs.py new file mode 100644 index 0000000000000000000000000000000000000000..928a3c1ab5ea3621485b3a0189529987850cf599 --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/datasets/flores_langs.py @@ -0,0 +1,203 @@ +flores_languages = [ + "ace_Arab", + "ace_Latn", + "acm_Arab", + "acq_Arab", + "aeb_Arab", + "afr_Latn", + "ajp_Arab", + "aka_Latn", + "amh_Ethi", + "apc_Arab", + "arb_Arab", + "ars_Arab", + "ary_Arab", + "arz_Arab", + "asm_Beng", + "ast_Latn", + "awa_Deva", + "ayr_Latn", + "azb_Arab", + "azj_Latn", + "bak_Cyrl", + "bam_Latn", + "ban_Latn", + "bel_Cyrl", + "bem_Latn", + "ben_Beng", + "bho_Deva", + "bjn_Arab", + "bjn_Latn", + "bod_Tibt", + "bos_Latn", + "bug_Latn", + "bul_Cyrl", + "cat_Latn", + "ceb_Latn", + "ces_Latn", + "cjk_Latn", + "ckb_Arab", + "crh_Latn", + "cym_Latn", + "dan_Latn", + "deu_Latn", + "dik_Latn", + "dyu_Latn", + "dzo_Tibt", + "eng_Latn", + "ell_Grek", + "epo_Latn", + "est_Latn", + "eus_Latn", + "ewe_Latn", + "fao_Latn", + "fij_Latn", + "fin_Latn", + "fon_Latn", + "fra_Latn", + "fur_Latn", + "fuv_Latn", + "gla_Latn", + "gle_Latn", + "glg_Latn", + "grn_Latn", + "guj_Gujr", + "hat_Latn", + "hau_Latn", + "heb_Hebr", + "hin_Deva", + "hne_Deva", + "hrv_Latn", + "hun_Latn", + "hye_Armn", + "ibo_Latn", + "ilo_Latn", + "ind_Latn", + "isl_Latn", + "ita_Latn", + "jav_Latn", + "jpn_Jpan", + "kab_Latn", + "kac_Latn", + "kam_Latn", + "kan_Knda", + "kas_Arab", + "kas_Deva", + "kat_Geor", + "knc_Arab", + "knc_Latn", + "kaz_Cyrl", + "kbp_Latn", + "kea_Latn", + "khm_Khmr", + "kik_Latn", + "kin_Latn", + "kir_Cyrl", + "kmb_Latn", + "kmr_Latn", + "kon_Latn", + "kor_Hang", + "lao_Laoo", + "lij_Latn", + "lim_Latn", + "lin_Latn", + "lit_Latn", + "lmo_Latn", + "ltg_Latn", + "ltz_Latn", + "lua_Latn", + "lug_Latn", + "luo_Latn", + "lus_Latn", + "lvs_Latn", + "mag_Deva", + "mai_Deva", + "mal_Mlym", + "mar_Deva", + "min_Latn", + "mkd_Cyrl", + "plt_Latn", + "mlt_Latn", + "mni_Beng", + "khk_Cyrl", + "mos_Latn", + "mri_Latn", + "mya_Mymr", + "nld_Latn", + "nno_Latn", + "nob_Latn", + "npi_Deva", + "nso_Latn", + "nus_Latn", + "nya_Latn", + "oci_Latn", + "gaz_Latn", + "ory_Orya", + "pag_Latn", + "pan_Guru", + "pap_Latn", + "pes_Arab", + "pol_Latn", + "por_Latn", + "prs_Arab", + "pbt_Arab", + "quy_Latn", + "ron_Latn", + "run_Latn", + "rus_Cyrl", + "sag_Latn", + "san_Deva", + "scn_Latn", + "shn_Mymr", + "sin_Sinh", + "slk_Latn", + "slv_Latn", + "smo_Latn", + "sna_Latn", + "snd_Arab", + "som_Latn", + "sot_Latn", + "spa_Latn", + "als_Latn", + "srd_Latn", + "srp_Cyrl", + "ssw_Latn", + "sun_Latn", + "swe_Latn", + "swh_Latn", + "szl_Latn", + "tam_Taml", + "tat_Cyrl", + "tel_Telu", + "tgk_Cyrl", + "tgl_Latn", + "tha_Thai", + "tir_Ethi", + "taq_Latn", + "taq_Tfng", + "tpi_Latn", + "tsn_Latn", + "tso_Latn", + "tuk_Latn", + "tum_Latn", + "tur_Latn", + "twi_Latn", + "tzm_Tfng", + "uig_Arab", + "ukr_Cyrl", + "umb_Latn", + "urd_Arab", + "uzn_Latn", + "vec_Latn", + "vie_Latn", + "war_Latn", + "wol_Latn", + "xho_Latn", + "ydd_Hebr", + "yor_Latn", + "yue_Hant", + "zho_Hans", + "zho_Hant", + "zsm_Latn", + "zul_Latn", +] diff --git a/perception_models/apps/pe/clip_benchmark/datasets/imagenetv2.py b/perception_models/apps/pe/clip_benchmark/datasets/imagenetv2.py new file mode 100644 index 0000000000000000000000000000000000000000..2bef6cbcfad33d2ea774cce9cd0f1380fcd4eec4 --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/datasets/imagenetv2.py @@ -0,0 +1,108 @@ +""" +Code from https://github.com/mlfoundations/wise-ft/blob/master/src/datasets/imagenetv2.py +Thanks to the authors of wise-ft +""" + +import pathlib +import shutil +import tarfile + +import requests +from PIL import Image +from torch.utils.data import DataLoader, Dataset +from torchvision.datasets import ImageFolder +from tqdm import tqdm + +URLS = { + "matched-frequency": "https://imagenetv2public.s3-us-west-2.amazonaws.com/imagenetv2-matched-frequency.tar.gz", + "threshold-0.7": "https://imagenetv2public.s3-us-west-2.amazonaws.com/imagenetv2-threshold0.7.tar.gz", + "top-images": "https://imagenetv2public.s3-us-west-2.amazonaws.com/imagenetv2-top-images.tar.gz", + "val": "https://imagenetv2public.s3-us-west-2.amazonaws.com/imagenet_validation.tar.gz", +} + +FNAMES = { + "matched-frequency": "imagenetv2-matched-frequency-format-val", + "threshold-0.7": "imagenetv2-threshold0.7-format-val", + "top-images": "imagenetv2-top-images-format-val", + "val": "imagenet_validation", +} + + +V2_DATASET_SIZE = 10000 +VAL_DATASET_SIZE = 50000 + + +class ImageNetValDataset(Dataset): + def __init__(self, transform=None, location="."): + self.dataset_root = pathlib.Path(f"{location}/imagenet_validation/") + self.tar_root = pathlib.Path(f"{location}/imagenet_validation.tar.gz") + self.fnames = list(self.dataset_root.glob("**/*.JPEG")) + self.transform = transform + if not self.dataset_root.exists() or len(self.fnames) != VAL_DATASET_SIZE: + if not self.tar_root.exists(): + print(f"Dataset imagenet-val not found on disk, downloading....") + response = requests.get(URLS["val"], stream=True) + total_size_in_bytes = int(response.headers.get("content-length", 0)) + block_size = 1024 # 1 Kibibyte + progress_bar = tqdm( + total=total_size_in_bytes, unit="iB", unit_scale=True + ) + with open(self.tar_root, "wb") as f: + for data in response.iter_content(block_size): + progress_bar.update(len(data)) + f.write(data) + progress_bar.close() + if total_size_in_bytes != 0 and progress_bar.n != total_size_in_bytes: + assert False, f"Downloading from {URLS[variant]} failed" + print("Extracting....") + tarfile.open(self.tar_root).extractall(f"{location}") + shutil.move(f"{location}/{FNAMES['val']}", self.dataset_root) + + self.dataset = ImageFolder(self.dataset_root) + + def __len__(self): + return len(self.dataset) + + def __getitem__(self, i): + img, label = self.dataset[i] + if self.transform is not None: + img = self.transform(img) + return img, label + + +class ImageNetV2Dataset(Dataset): + def __init__(self, variant="matched-frequency", transform=None, location="."): + self.dataset_root = pathlib.Path(f"{location}/ImageNetV2-{variant}/") + self.tar_root = pathlib.Path(f"{location}/ImageNetV2-{variant}.tar.gz") + self.fnames = list(self.dataset_root.glob("**/*.jpeg")) + self.transform = transform + assert variant in URLS, f"unknown V2 Variant: {variant}" + if not self.dataset_root.exists() or len(self.fnames) != V2_DATASET_SIZE: + if not self.tar_root.exists(): + print(f"Dataset {variant} not found on disk, downloading....") + response = requests.get(URLS[variant], stream=True) + total_size_in_bytes = int(response.headers.get("content-length", 0)) + block_size = 1024 # 1 Kibibyte + progress_bar = tqdm( + total=total_size_in_bytes, unit="iB", unit_scale=True + ) + with open(self.tar_root, "wb") as f: + for data in response.iter_content(block_size): + progress_bar.update(len(data)) + f.write(data) + progress_bar.close() + if total_size_in_bytes != 0 and progress_bar.n != total_size_in_bytes: + assert False, f"Downloading from {URLS[variant]} failed" + print("Extracting....") + tarfile.open(self.tar_root).extractall(f"{location}") + shutil.move(f"{location}/{FNAMES[variant]}", self.dataset_root) + self.fnames = list(self.dataset_root.glob("**/*.jpeg")) + + def __len__(self): + return len(self.fnames) + + def __getitem__(self, i): + img, label = Image.open(self.fnames[i]), int(self.fnames[i].parent.name) + if self.transform is not None: + img = self.transform(img) + return img, label diff --git a/perception_models/apps/pe/clip_benchmark/datasets/kitti.py b/perception_models/apps/pe/clip_benchmark/datasets/kitti.py new file mode 100644 index 0000000000000000000000000000000000000000..4f8cbb06bd03327d9f7ee1b93b0357c5543819d0 --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/datasets/kitti.py @@ -0,0 +1,208 @@ +# coding=utf-8 +# Copyright 2019 Google LLC. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Implements Kitti data class.""" + +from __future__ import absolute_import, division, print_function + +import numpy as np +import task_adaptation.data.base as base +import tensorflow.compat.v1 as tf +import tensorflow_datasets as tfds +from task_adaptation.registry import Registry + + +def _count_all_pp(x): + """Count all objects.""" + # Count distribution (thresholded at 15): + + label = tf.math.minimum(tf.size(x["objects"]["type"]) - 1, 8) + return {"image": x["image"], "label": label} + + +def _count_vehicles_pp(x): + """Counting vehicles.""" + # Label distribution: + + vehicles = tf.where(x["objects"]["type"] < 3) # Car, Van, Truck. + # Cap at 3. + label = tf.math.minimum(tf.size(vehicles), 3) + return {"image": x["image"], "label": label} + + +def _count_left_pp(x): + """Count objects on the left hand side of the camera.""" + # Count distribution (thresholded at 15): + + # Location feature contains (x, y, z) in meters w.r.t. the camera. + objects_on_left = tf.where(x["objects"]["location"][:, 0] < 0) + label = tf.math.minimum(tf.size(objects_on_left), 8) + return {"image": x["image"], "label": label} + + +def _count_far_pp(x): + """Counts objects far from the camera.""" + # Threshold removes ~half of the objects. + # Count distribution (thresholded at 15): + + # Location feature contains (x, y, z) in meters w.r.t. the camera. + distant_objects = tf.where(x["objects"]["location"][:, 2] >= 25) + label = tf.math.minimum(tf.size(distant_objects), 8) + return {"image": x["image"], "label": label} + + +def _count_near_pp(x): + """Counts objects close to the camera.""" + # Threshold removes ~half of the objects. + # Count distribution: + + # Location feature contains (x, y, z) in meters w.r.t. the camera. + close_objects = tf.where(x["objects"]["location"][:, 2] < 25) + label = tf.math.minimum(tf.size(close_objects), 8) + return {"image": x["image"], "label": label} + + +def _closest_object_distance_pp(x): + """Predict the distance to the closest object.""" + # Label distribution: + + # Location feature contains (x, y, z) in meters w.r.t. the camera. + dist = tf.reduce_min(x["objects"]["location"][:, 2]) + thrs = np.array([-100, 5.6, 8.4, 13.4, 23.4]) + label = tf.reduce_max(tf.where((thrs - dist) < 0)) + return {"image": x["image"], "label": label} + + +def _closest_vehicle_distance_pp(x): + """Predict the distance to the closest vehicle.""" + # Label distribution: + + # Location feature contains (x, y, z) in meters w.r.t. the camera. + vehicles = tf.where(x["objects"]["type"] < 3) # Car, Van, Truck. + vehicle_z = tf.gather(params=x["objects"]["location"][:, 2], indices=vehicles) + vehicle_z = tf.concat([vehicle_z, tf.constant([[1000.0]])], axis=0) + dist = tf.reduce_min(vehicle_z) + # Results in a uniform distribution over three distances, plus one class for + # "no vehicle". + thrs = np.array([-100.0, 8.0, 20.0, 999.0]) + label = tf.reduce_max(tf.where((thrs - dist) < 0)) + return {"image": x["image"], "label": label} + + +def _closest_object_x_location_pp(x): + """Predict the absolute x position of the closest object.""" + # Label distribution: + + # Location feature contains (x, y, z) in meters w.r.t. the camera. + idx = tf.math.argmin(x["objects"]["location"][:, 2]) + xloc = x["objects"]["location"][idx, 0] + thrs = np.array([-100, -6.4, -3.5, 0.0, 3.3, 23.9]) + label = tf.reduce_max(tf.where((thrs - xloc) < 0)) + return {"image": x["image"], "label": label} + + +_TASK_DICT = { + "count_all": { + "preprocess_fn": _count_all_pp, + "num_classes": 16, + }, + "count_left": { + "preprocess_fn": _count_left_pp, + "num_classes": 16, + }, + "count_far": { + "preprocess_fn": _count_far_pp, + "num_classes": 16, + }, + "count_near": { + "preprocess_fn": _count_near_pp, + "num_classes": 16, + }, + "closest_object_distance": { + "preprocess_fn": _closest_object_distance_pp, + "num_classes": 5, + }, + "closest_object_x_location": { + "preprocess_fn": _closest_object_x_location_pp, + "num_classes": 5, + }, + "count_vehicles": { + "preprocess_fn": _count_vehicles_pp, + "num_classes": 4, + }, + "closest_vehicle_distance": { + "preprocess_fn": _closest_vehicle_distance_pp, + "num_classes": 4, + }, +} + + +@Registry.register("data.kitti", "class") +class KittiData(base.ImageTfdsData): + """Provides Kitti dataset. + + Six tasks are supported: + 1. Count the number of objects. + 2. Count the number of objects on the left hand side of the camera. + 3. Count the number of objects in the foreground. + 4. Count the number of objects in the background. + 5. Predict the distance of the closest object. + 6. Predict the x-location (w.r.t. the camera) of the closest object. + """ + + def __init__(self, task, data_dir=None): + + if task not in _TASK_DICT: + raise ValueError("Unknown task: %s" % task) + + dataset_builder = tfds.builder("kitti:3.3.0", data_dir=data_dir) + dataset_builder.download_and_prepare() + + tfds_splits = { + "train": "train", + "val": "validation", + "trainval": "train+validation", + "test": "test", + "train800": "train[:800]", + "val200": "validation[:200]", + "train800val200": "train[:800]+validation[:200]", + } + + # Example counts are retrieved from the tensorflow dataset info. + train_count = dataset_builder.info.splits[tfds.Split.TRAIN].num_examples + val_count = dataset_builder.info.splits[tfds.Split.VALIDATION].num_examples + test_count = dataset_builder.info.splits[tfds.Split.TEST].num_examples + # Creates a dict with example counts for each split. + num_samples_splits = { + "train": train_count, + "val": val_count, + "trainval": train_count + val_count, + "test": test_count, + "train800": 800, + "val200": 200, + "train800val200": 1000, + } + + task = _TASK_DICT[task] + base_preprocess_fn = task["preprocess_fn"] + super(KittiData, self).__init__( + dataset_builder=dataset_builder, + tfds_splits=tfds_splits, + num_samples_splits=num_samples_splits, + num_preprocessing_threads=400, + shuffle_buffer_size=10000, + base_preprocess_fn=base_preprocess_fn, + num_classes=task["num_classes"], + ) diff --git a/perception_models/apps/pe/clip_benchmark/datasets/multilingual_mscoco.py b/perception_models/apps/pe/clip_benchmark/datasets/multilingual_mscoco.py new file mode 100644 index 0000000000000000000000000000000000000000..49d11500e50a9e1f6fbd86748f6f39eddd586a25 --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/datasets/multilingual_mscoco.py @@ -0,0 +1,121 @@ +import codecs +import json +import os +from subprocess import call + +import requests +from PIL import Image +from torchvision.datasets import VisionDataset + +GITHUB_DATA_PATH = "https://raw.githubusercontent.com/adobe-research/Cross-lingual-Test-Dataset-XTD10/main/XTD10/" +GITHUB_DATA_PATH_DE_FR = "https://raw.githubusercontent.com/adobe-research/Cross-lingual-Test-Dataset-XTD10/main/MIC/" +GITHUB_DATA_PATH_JP = "https://raw.githubusercontent.com/adobe-research/Cross-lingual-Test-Dataset-XTD10/main/STAIR/" +SUPPORTED_LANGUAGES = ["es", "it", "ko", "pl", "ru", "tr", "zh", "en", "de", "fr", "jp"] + +IMAGE_INDEX_FILENAME = "test_image_names.txt" + +CAPTIONS_FILENAME_TEMPLATE = "test_1kcaptions_{}.txt" +OUTPUT_FILENAME_TEMPLATE = "multilingual_mscoco_captions-{}.json" + +IMAGES_DOWNLOAD_URL = "https://nllb-data.com/test/xtd10/images.tar.gz" + + +class Multilingual_MSCOCO(VisionDataset): + def __init__(self, root, ann_file, transform=None, target_transform=None): + super().__init__(root, transform=transform, target_transform=target_transform) + self.ann_file = os.path.expanduser(ann_file) + with codecs.open(ann_file, "r", encoding="utf-8") as fp: + data = json.load(fp) + self.data = [ + (img_path, txt) + for img_path, txt in zip(data["image_paths"], data["annotations"]) + ] + + def __getitem__(self, index): + img, captions = self.data[index] + + # Image + img = Image.open(img).convert("RGB") + if self.transform is not None: + img = self.transform(img) + + # Captions + target = [ + captions, + ] + if self.target_transform is not None: + target = self.target_transform(target) + + return img, target + + def __len__(self) -> int: + return len(self.data) + + +def _get_lines(url): + response = requests.get(url, timeout=30) + return response.text.splitlines() + + +def _download_images(out_path): + os.makedirs(out_path, exist_ok=True) + print("Downloading images") + call(f"wget {IMAGES_DOWNLOAD_URL} -O images.tar.gz", shell=True) + call(f"tar -xzf images.tar.gz -C {out_path}", shell=True) + call("rm images.tar.gz", shell=True) + + +def create_annotation_file(root, lang_code): + if lang_code not in SUPPORTED_LANGUAGES: + raise ValueError( + f"Language code {lang_code} not supported. Supported languages are {SUPPORTED_LANGUAGES}" + ) + data_dir = os.path.join(root, "multilingual_mscoco") + if not os.path.exists(data_dir): + _download_images(data_dir) + images_dir = os.path.join(data_dir, "images") + print("Downloading multilingual_ms_coco index file") + download_path = os.path.join(GITHUB_DATA_PATH, IMAGE_INDEX_FILENAME) + target_images = _get_lines(download_path) + + print("Downloading multilingual_ms_coco captions:", lang_code) + captions_path = GITHUB_DATA_PATH + if lang_code in ["de", "fr"]: + captions_path = GITHUB_DATA_PATH_DE_FR + elif lang_code == "jp": + captions_path = GITHUB_DATA_PATH_JP + download_path = os.path.join( + captions_path, CAPTIONS_FILENAME_TEMPLATE.format(lang_code) + ) + target_captions = _get_lines(download_path) + + number_of_missing_images = 0 + valid_images, valid_annotations, valid_indicies = [], [], [] + for i, (img, txt) in enumerate(zip(target_images, target_captions)): + image_path = os.path.join(images_dir, img) + if not os.path.exists(image_path): + print("Missing image file", img) + number_of_missing_images += 1 + continue + + valid_images.append(image_path) + valid_annotations.append(txt) + valid_indicies.append(i) + + if number_of_missing_images > 0: + print(f"*** WARNING *** missing {number_of_missing_images} files.") + + with codecs.open( + os.path.join(root, OUTPUT_FILENAME_TEMPLATE.format(lang_code)), + "w", + encoding="utf-8", + ) as fp: + json.dump( + { + "image_paths": valid_images, + "annotations": valid_annotations, + "indicies": valid_indicies, + }, + fp, + ensure_ascii=False, + ) diff --git a/perception_models/apps/pe/clip_benchmark/datasets/objectnet.py b/perception_models/apps/pe/clip_benchmark/datasets/objectnet.py new file mode 100644 index 0000000000000000000000000000000000000000..d2eb4e6033b69a28a347db22f4a2b5701e5e5bd2 --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/datasets/objectnet.py @@ -0,0 +1,83 @@ +""" +Code adapted from https://github.com/mlfoundations/wise-ft/blob/master/src/datasets/objectnet.py +Thanks to the authors of wise-ft +""" + +import json +import os +from pathlib import Path + +import numpy as np +import PIL +import torch +from torchvision import datasets +from torchvision.transforms import Compose + + +def get_metadata(folder): + metadata = Path(folder) + + with open(metadata / "folder_to_objectnet_label.json", "r") as f: + folder_map = json.load(f) + folder_map = {v: k for k, v in folder_map.items()} + with open(metadata / "objectnet_to_imagenet_1k.json", "r") as f: + objectnet_map = json.load(f) + + with open(metadata / "pytorch_to_imagenet_2012_id.json", "r") as f: + pytorch_map = json.load(f) + pytorch_map = {v: k for k, v in pytorch_map.items()} + + with open(metadata / "imagenet_to_label_2012_v2", "r") as f: + imagenet_map = {v.strip(): str(pytorch_map[i]) for i, v in enumerate(f)} + + folder_to_ids, class_sublist = {}, [] + classnames = [] + for objectnet_name, imagenet_names in objectnet_map.items(): + imagenet_names = imagenet_names.split("; ") + imagenet_ids = [ + int(imagenet_map[imagenet_name]) for imagenet_name in imagenet_names + ] + class_sublist.extend(imagenet_ids) + folder_to_ids[folder_map[objectnet_name]] = imagenet_ids + + class_sublist = sorted(class_sublist) + class_sublist_mask = [(i in class_sublist) for i in range(1000)] + classname_map = {v: k for k, v in folder_map.items()} + return class_sublist, class_sublist_mask, folder_to_ids, classname_map + + +class ObjectNetDataset(datasets.ImageFolder): + + def __init__(self, root, transform): + ( + self._class_sublist, + self.class_sublist_mask, + self.folders_to_ids, + self.classname_map, + ) = get_metadata(root) + subdir = os.path.join(root, "objectnet-1.0", "images") + label_map = { + name: idx + for idx, name in enumerate(sorted(list(self.folders_to_ids.keys()))) + } + self.label_map = label_map + super().__init__(subdir, transform=transform) + self.samples = [ + d + for d in self.samples + if os.path.basename(os.path.dirname(d[0])) in self.label_map + ] + self.imgs = self.samples + self.classes = sorted(list(self.folders_to_ids.keys())) + self.classes = [self.classname_map[c].lower() for c in self.classes] + + def __len__(self): + return len(self.samples) + + def __getitem__(self, index): + path, target = self.samples[index] + sample = self.loader(path) + if self.transform is not None: + sample = self.transform(sample) + label = os.path.basename(os.path.dirname(path)) + return sample, self.label_map[label] diff --git a/perception_models/apps/pe/clip_benchmark/datasets/pos_neg_caption_dataset.py b/perception_models/apps/pe/clip_benchmark/datasets/pos_neg_caption_dataset.py new file mode 100644 index 0000000000000000000000000000000000000000..c17a889d6dd0f3f5b78fdd17b32d263954ceeaa6 --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/datasets/pos_neg_caption_dataset.py @@ -0,0 +1,38 @@ +import json +import os + +from PIL import Image +from torch.utils.data import Dataset + + +class PosNegCaptionDataset(Dataset): + + def __init__(self, root, ann_file, transform=None, crop_images=False): + self.root = root + self.ann = json.load(open(ann_file)) + self.transform = transform + self.crop_images = crop_images + self.idx_strings = list(self.ann.keys()) # NOTE : indices may be non-contiguous + + def __getitem__(self, idx): + idx_str = self.idx_strings[idx] + data = self.ann[idx_str] + img = Image.open(os.path.join(self.root, data["filename"])) + if self.crop_images: + img = img.crop( + ( + data["bbox_x"], + data["bbox_y"], + data["bbox_x"] + data["bbox_width"], + data["bbox_y"] + data["bbox_height"], + ) + ) + if self.transform is not None: + img = self.transform(img) + caption = data["caption"] + negative_caption = data["negative_caption"] + + return img, [caption, negative_caption] + + def __len__(self): + return len(self.ann) diff --git a/perception_models/apps/pe/clip_benchmark/datasets/tfds.py b/perception_models/apps/pe/clip_benchmark/datasets/tfds.py new file mode 100644 index 0000000000000000000000000000000000000000..dc18b5deb1dc87d77f9965f6e8f3ab2a01536e6b --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/datasets/tfds.py @@ -0,0 +1,67 @@ +import torch +from PIL import Image + + +def download_tfds_dataset(name, data_dir=None): + import tensorflow_datasets as tfds + import timm + + builder = tfds.builder(name, data_dir=data_dir) + builder.download_and_prepare() + + +def disable_gpus_on_tensorflow(): + import tensorflow as tf + + tf.config.set_visible_devices([], "GPU") + + +class VTABIterableDataset(torch.utils.data.IterableDataset): + + def __init__( + self, + tfds_dataset, + split="test", + input_name="image", + label_name="label", + input_mode="RGB", + transform=None, + target_transform=None, + classes=None, + ): + self.tfds_dataset = tfds_dataset + self.input_name = input_name + self.label_name = label_name + self.transform = transform + self.target_transform = target_transform + self.input_mode = input_mode + self.num_examples = tfds_dataset.get_num_samples(split) + self.split = split + if classes is None: + self.classes = tfds_dataset._dataset_builder.info.features["label"].names + else: + self.classes = classes + + def __iter__(self): + worker_info = torch.utils.data.get_worker_info() + iterator = self.tfds_dataset.get_tf_data( + self.split, batch_size=1, epochs=1, for_eval=True + ) + if worker_info is not None: + iterator = iterator.shard( + index=worker_info.id, num_shards=worker_info.num_workers + ) + nb = 0 + for data in iterator: + inputs = data[self.input_name].numpy() + labels = data[self.label_name].numpy() + for input, label in zip(inputs, labels): + input = Image.fromarray(input, mode=self.input_mode) + if self.transform is not None: + input = self.transform(input) + if self.target_transform is not None: + label = self.target_transform(label) + yield input, label + + def __len__(self): + return self.num_examples diff --git a/perception_models/apps/pe/clip_benchmark/datasets/video_classification_dataset.py b/perception_models/apps/pe/clip_benchmark/datasets/video_classification_dataset.py new file mode 100644 index 0000000000000000000000000000000000000000..bcd5d37f5af8494144714c3c3e18233479218ff3 --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/datasets/video_classification_dataset.py @@ -0,0 +1,119 @@ +import os +import random + +import cv2 +import decord +import torch +from PIL import Image +from torch.utils.data import Dataset + + +class VideoClassificationDataset(Dataset): + def __init__(self, dataset_dir_path, task_config, preprocessor, num_frames=8): + self.dataset_dir_path = dataset_dir_path + self.labels_txt = task_config["labels"] + self.media_dir_path = os.path.join(dataset_dir_path, task_config["media"]) + self.class_ids, self.classes = self.get_class_info() + + self.media_paths = [] + self.labels = [] + self.label_ids = [] + + for j, (class_id, class_name) in enumerate(zip(self.class_ids, self.classes)): + class_dir_path = os.path.join(self.media_dir_path, class_id) + for i, video_file_name in enumerate(os.listdir(class_dir_path)): + video_path = os.path.join(class_dir_path, video_file_name) + self.media_paths.append(video_path) + self.labels.append(class_name) + self.label_ids.append(j) + + self.preprocessor = preprocessor + self.num_frames = num_frames + + def get_class_info(self): + class_ids = [ + dir_name + for dir_name in os.listdir(self.media_dir_path) + if os.path.isdir(os.path.join(self.media_dir_path, dir_name)) + ] + + if self.labels_txt: + labels_txt_path = os.path.join(self.dataset_dir_path, self.labels_txt) + id_to_class_name = {} + with open(labels_txt_path, "r") as f: + for line in f: + id, class_name = line.strip().split(",") + id_to_class_name[id] = class_name + class_names = [id_to_class_name[id] for id in class_ids] + else: + class_names = class_ids + + def clean_label(label: str) -> str: + """ + Return a label without spaces or parenthesis + """ + for c in "()": + label = label.replace(c, "") + return label.strip("_") + + class_names = [clean_label(label) for label in class_names] + + return class_ids, class_names + + def __len__(self): + return len(self.media_paths) + + def __getitem__(self, index): + while True: + media_path = self.media_paths[index] + class_name = self.labels[index] + class_id = self.label_ids[index] + + try: + images = self._load_video(media_path) + + images = [ + ( + self.preprocessor(image.convert("RGB")) + if image.mode == "L" + else self.preprocessor(image) + ) + for image in images + ] + break + except Exception as e: + print(f"{e}, skipping {media_path}.") + index = random.randint(0, len(self.media_paths) - 1) + + # Returns a list of images and one class_id. The model will need to aggregate across the list of images to make a prediction. + return images, class_id + + def _load_video(self, media_path): + vr = decord.VideoReader(media_path) + total_frames = len(vr) + if self.num_frames == 1: + frame_indices = [total_frames // 2] + else: + frame_indices = [ + int(i * (total_frames - 1) / (self.num_frames - 1)) + for i in range(self.num_frames) + ] + + try: + images = vr.get_batch(frame_indices).asnumpy() + except Exception as e: + cap = cv2.VideoCapture(media_path) + images = [] + for pos in frame_indices: + cap.set(cv2.CAP_PROP_POS_FRAMES, pos) + ret, frame = cap.read() + if ret: + # Convert the frame from BGR to RGB + rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) + images.append(rgb_frame) + else: + break + + images = [Image.fromarray(image) for image in images] + + return images diff --git a/perception_models/apps/pe/clip_benchmark/datasets/video_retrieval_dataset.py b/perception_models/apps/pe/clip_benchmark/datasets/video_retrieval_dataset.py new file mode 100644 index 0000000000000000000000000000000000000000..c499475725849755cc37b39d2c3d5aabf06f5f49 --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/datasets/video_retrieval_dataset.py @@ -0,0 +1,84 @@ +import os +import random + +import cv2 +import decord +import pandas as pd +import torch +from PIL import Image +from torch.utils.data import Dataset + + +class VideoRetrievalDataset(Dataset): + def __init__( + self, + csv_path, + dataset_dir, + preprocessor, + video_ext="mp4", + num_frames=8, + multi_sent=False, + ): + self.data = pd.read_csv(csv_path) + self.dataset_dir = dataset_dir + self.video_ext = video_ext + + self.preprocessor = preprocessor + self.num_frames = num_frames + self.multi_sent = multi_sent + + def __len__(self): + return len(self.data) + + def __getitem__(self, index): + video_id = self.data["video_id"].values[index] + sentences = self.data["sentence"].values[index] + if self.multi_sent: + sentences = sentences.split("@") + else: + sentences = [sentences] + video_path = os.path.join( + self.dataset_dir, "{}.{}".format(video_id, self.video_ext) + ) + + images = self._load_video(video_path) + + images = [ + ( + self.preprocessor(image.convert("RGB")) + if image.mode == "L" + else self.preprocessor(image) + ) + for image in images + ] + + return images, sentences + + def _load_video(self, media_path): + vr = decord.VideoReader(media_path) + total_frames = len(vr) + if self.num_frames == 1: + frame_indices = [total_frames // 2] + else: + frame_indices = [ + int(i * (total_frames - 1) / (self.num_frames - 1)) + for i in range(self.num_frames) + ] + try: + images = vr.get_batch(frame_indices).asnumpy() + except Exception as e: + cap = cv2.VideoCapture(media_path) + images = [] + for pos in frame_indices: + cap.set(cv2.CAP_PROP_POS_FRAMES, pos) + ret, frame = cap.read() + if ret: + # Convert the frame from BGR to RGB + rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) + images.append(rgb_frame) + else: + break + + images = [Image.fromarray(image) for image in images] + + return images diff --git a/perception_models/apps/pe/clip_benchmark/datasets/voc2007.py b/perception_models/apps/pe/clip_benchmark/datasets/voc2007.py new file mode 100644 index 0000000000000000000000000000000000000000..a771d75721be7a4f0269e5ffc2bc43a0d1a605e7 --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/datasets/voc2007.py @@ -0,0 +1,290 @@ +# Code from https://github.com/SsnL/dataset-distillation/blob/master/datasets/pascal_voc.py , thanks to the authors +"""Dataset setting and data loader for PASCAL VOC 2007 as a classification task. + +Modified from +https://github.com/Cadene/pretrained-models.pytorch/blob/56aa8c921819d14fb36d7248ab71e191b37cb146/pretrainedmodels/datasets/voc.py +""" + +import os +import os.path +import tarfile +import xml.etree.ElementTree as ET +from urllib.parse import urlparse + +import torch +import torch.utils.data as data +import torchvision +from PIL import Image + +object_categories = [ + "aeroplane", + "bicycle", + "bird", + "boat", + "bottle", + "bus", + "car", + "cat", + "chair", + "cow", + "diningtable", + "dog", + "horse", + "motorbike", + "person", + "pottedplant", + "sheep", + "sofa", + "train", + "tvmonitor", +] + +category_to_idx = {c: i for i, c in enumerate(object_categories)} + +urls = { + "devkit": "http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCdevkit_08-Jun-2007.tar", + "trainval_2007": "http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtrainval_06-Nov-2007.tar", + "test_images_2007": "http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtest_06-Nov-2007.tar", + "test_anno_2007": "http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtestnoimgs_06-Nov-2007.tar", +} + + +def download_url(url, path): + root, filename = os.path.split(path) + torchvision.datasets.utils.download_url(url, root=root, filename=filename, md5=None) + + +def download_voc2007(root): + path_devkit = os.path.join(root, "VOCdevkit") + path_images = os.path.join(root, "VOCdevkit", "VOC2007", "JPEGImages") + tmpdir = os.path.join(root, "tmp") + + # create directory + if not os.path.exists(root): + os.makedirs(root) + + if not os.path.exists(path_devkit): + + if not os.path.exists(tmpdir): + os.makedirs(tmpdir) + + parts = urlparse(urls["devkit"]) + filename = os.path.basename(parts.path) + cached_file = os.path.join(tmpdir, filename) + + if not os.path.exists(cached_file): + download_url(urls["devkit"], cached_file) + + # extract file + print( + "[dataset] Extracting tar file {file} to {path}".format( + file=cached_file, path=root + ) + ) + cwd = os.getcwd() + tar = tarfile.open(cached_file, "r") + os.chdir(root) + tar.extractall() + tar.close() + os.chdir(cwd) + print("[dataset] Done!") + + # train/val images/annotations + if not os.path.exists(path_images): + + # download train/val images/annotations + parts = urlparse(urls["trainval_2007"]) + filename = os.path.basename(parts.path) + cached_file = os.path.join(tmpdir, filename) + + if not os.path.exists(cached_file): + download_url(urls["trainval_2007"], cached_file) + + # extract file + print( + "[dataset] Extracting tar file {file} to {path}".format( + file=cached_file, path=root + ) + ) + cwd = os.getcwd() + tar = tarfile.open(cached_file, "r") + os.chdir(root) + tar.extractall() + tar.close() + os.chdir(cwd) + print("[dataset] Done!") + + # test annotations + test_anno = os.path.join(path_devkit, "VOC2007/ImageSets/Main/aeroplane_test.txt") + if not os.path.exists(test_anno): + + # download test annotations + parts = urlparse(urls["test_images_2007"]) + filename = os.path.basename(parts.path) + cached_file = os.path.join(tmpdir, filename) + + if not os.path.exists(cached_file): + download_url(urls["test_images_2007"], cached_file) + + # extract file + print( + "[dataset] Extracting tar file {file} to {path}".format( + file=cached_file, path=root + ) + ) + cwd = os.getcwd() + tar = tarfile.open(cached_file, "r") + os.chdir(root) + tar.extractall() + tar.close() + os.chdir(cwd) + print("[dataset] Done!") + + # test images + test_image = os.path.join(path_devkit, "VOC2007/JPEGImages/000001.jpg") + if not os.path.exists(test_image): + + # download test images + parts = urlparse(urls["test_anno_2007"]) + filename = os.path.basename(parts.path) + cached_file = os.path.join(tmpdir, filename) + + if not os.path.exists(cached_file): + download_url(urls["test_anno_2007"], cached_file) + + # extract file + print( + "[dataset] Extracting tar file {file} to {path}".format( + file=cached_file, path=root + ) + ) + cwd = os.getcwd() + tar = tarfile.open(cached_file, "r") + os.chdir(root) + tar.extractall() + tar.close() + os.chdir(cwd) + print("[dataset] Done!") + + +def read_split(root, dataset, split): + base_path = os.path.join(root, "VOCdevkit", dataset, "ImageSets", "Main") + filename = os.path.join(base_path, object_categories[0] + "_" + split + ".txt") + + with open(filename, "r") as f: + paths = [] + for line in f.readlines(): + line = line.strip().split() + if len(line) > 0: + assert len(line) == 2 + paths.append(line[0]) + + return tuple(paths) + + +def read_bndbox(root, dataset, paths): + xml_base = os.path.join(root, "VOCdevkit", dataset, "Annotations") + instances = [] + for path in paths: + xml = ET.parse(os.path.join(xml_base, path + ".xml")) + for obj in xml.findall("object"): + c = obj[0] + assert c.tag == "name", c.tag + c = category_to_idx[c.text] + bndbox = obj.find("bndbox") + xmin = int(bndbox[0].text) # left + ymin = int(bndbox[1].text) # top + xmax = int(bndbox[2].text) # right + ymax = int(bndbox[3].text) # bottom + instances.append((path, (xmin, ymin, xmax, ymax), c)) + return instances + + +class PASCALVoc2007(data.Dataset): + """ + Multi-label classification problem for voc2007 + labels are of one hot of shape (C,), denoting the presence/absence + of each class in each image, where C is the number of classes. + """ + + def __init__( + self, root, set, transform=None, download=False, target_transform=None + ): + self.root = root + self.path_devkit = os.path.join(root, "VOCdevkit") + self.path_images = os.path.join(root, "VOCdevkit", "VOC2007", "JPEGImages") + self.transform = transform + self.target_transform = target_transform + + # download dataset + if download: + download_voc2007(self.root) + + paths = read_split(self.root, "VOC2007", set) + bndboxes = read_bndbox(self.root, "VOC2007", paths) + labels = torch.zeros(len(paths), len(object_categories)) + path_index = {} + for i, p in enumerate(paths): + path_index[p] = i + for path, bbox, c in bndboxes: + labels[path_index[path], c] = 1 + self.labels = labels + self.classes = object_categories + self.paths = paths + + def __getitem__(self, index): + path = self.paths[index] + img = Image.open(os.path.join(self.path_images, path + ".jpg")).convert("RGB") + target = self.labels[index] + if self.transform is not None: + img = self.transform(img) + if self.target_transform is not None: + target = self.target_transform(target) + return img, target + + def __len__(self): + return len(self.paths) + + +class PASCALVoc2007Cropped(data.Dataset): + """ + voc2007 is originally object detection and multi-label. + In this version, we just convert it to single-label per image classification + problem by looping over bounding boxes in the dataset and cropping the relevant + object. + """ + + def __init__( + self, root, set, transform=None, download=False, target_transform=None + ): + self.root = root + self.path_devkit = os.path.join(root, "VOCdevkit") + self.path_images = os.path.join(root, "VOCdevkit", "VOC2007", "JPEGImages") + self.transform = transform + self.target_transform = target_transform + + # download dataset + if download: + download_voc2007(self.root) + + paths = read_split(self.root, "VOC2007", set) + self.bndboxes = read_bndbox(self.root, "VOC2007", paths) + self.classes = object_categories + + print( + "[dataset] VOC 2007 classification set=%s number of classes=%d number of bndboxes=%d" + % (set, len(self.classes), len(self.bndboxes)) + ) + + def __getitem__(self, index): + path, crop, target = self.bndboxes[index] + img = Image.open(os.path.join(self.path_images, path + ".jpg")).convert("RGB") + img = img.crop(crop) + if self.transform is not None: + img = self.transform(img) + if self.target_transform is not None: + target = self.target_transform(target) + return img, target + + def __len__(self): + return len(self.bndboxes) diff --git a/perception_models/apps/pe/clip_benchmark/datasets/winoground.py b/perception_models/apps/pe/clip_benchmark/datasets/winoground.py new file mode 100644 index 0000000000000000000000000000000000000000..9095d99b53fab352d133ab2dc696e2d548116022 --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/datasets/winoground.py @@ -0,0 +1,33 @@ +import json +import os + +import torch +from PIL import Image +from torch.utils.data import Dataset + + +class WinoGround(Dataset): + + def __init__(self, root=".", transform=None): + from datasets import load_dataset + + self.ds = load_dataset("facebook/winoground", cache_dir=root)["test"] + self.transform = transform + + def __getitem__(self, idx): + data = self.ds[idx] + img0 = data["image_0"] + img1 = data["image_1"] + cap0 = data["caption_0"] + cap1 = data["caption_1"] + if self.transform is not None: + img0 = self.transform(img0) + img1 = self.transform(img1) + imgs = torch.stack([img0, img1]) + else: + imgs = [img0, img1] + caps = [cap0, cap1] + return imgs, caps + + def __len__(self): + return len(self.ds) diff --git a/perception_models/apps/pe/clip_benchmark/datasets/xtd200.py b/perception_models/apps/pe/clip_benchmark/datasets/xtd200.py new file mode 100644 index 0000000000000000000000000000000000000000..935ac5a67539c7bffba6f0be527227af56a9c995 --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/datasets/xtd200.py @@ -0,0 +1,119 @@ +import codecs +import json +import os +from subprocess import call + +import requests +from PIL import Image +from torchvision.datasets import VisionDataset + +from .flores_langs import flores_languages + +GITHUB_DATA_PATH = ( + "https://raw.githubusercontent.com/visheratin/nllb-clip/main/data/xtd200/" +) +SUPPORTED_LANGUAGES = flores_languages + +IMAGE_INDEX_FILENAME = "test_image_names.txt" + +CAPTIONS_FILENAME_TEMPLATE = "{}.txt" +OUTPUT_FILENAME_TEMPLATE = "xtd200-{}.json" + +IMAGES_DOWNLOAD_URL = "https://nllb-data.com/test/xtd10/images.tar.gz" + + +class XTD200(VisionDataset): + def __init__(self, root, ann_file, transform=None, target_transform=None): + super().__init__(root, transform=transform, target_transform=target_transform) + self.ann_file = os.path.expanduser(ann_file) + with codecs.open(ann_file, "r", encoding="utf-8") as fp: + data = json.load(fp) + self.data = [ + (img_path, txt) + for img_path, txt in zip(data["image_paths"], data["annotations"]) + ] + + def __getitem__(self, index): + img, captions = self.data[index] + + # Image + img = Image.open(img).convert("RGB") + if self.transform is not None: + img = self.transform(img) + + # Captions + target = [ + captions, + ] + if self.target_transform is not None: + target = self.target_transform(target) + + return img, target + + def __len__(self) -> int: + return len(self.data) + + +def _get_lines(url): + response = requests.get(url, timeout=30) + return response.text.splitlines() + + +def _download_images(out_path): + os.makedirs(out_path, exist_ok=True) + print("Downloading images") + call(f"wget {IMAGES_DOWNLOAD_URL} -O images.tar.gz", shell=True) + call(f"tar -xzf images.tar.gz -C {out_path}", shell=True) + call("rm images.tar.gz", shell=True) + + +def create_annotation_file(root, lang_code): + if lang_code not in SUPPORTED_LANGUAGES: + raise ValueError( + f"Language code {lang_code} not supported. Supported languages are {SUPPORTED_LANGUAGES}" + ) + data_dir = os.path.join(root, "xtd200") + if not os.path.exists(data_dir): + _download_images(data_dir) + images_dir = os.path.join(data_dir, "images") + print("Downloading xtd200 index file") + download_path = os.path.join(GITHUB_DATA_PATH, IMAGE_INDEX_FILENAME) + target_images = _get_lines(download_path) + + print("Downloading xtd200 captions:", lang_code) + captions_path = GITHUB_DATA_PATH + download_path = os.path.join( + captions_path, CAPTIONS_FILENAME_TEMPLATE.format(lang_code) + ) + target_captions = _get_lines(download_path) + + number_of_missing_images = 0 + valid_images, valid_annotations, valid_indicies = [], [], [] + for i, (img, txt) in enumerate(zip(target_images, target_captions)): + image_path = os.path.join(images_dir, img) + if not os.path.exists(image_path): + print("Missing image file", img) + number_of_missing_images += 1 + continue + + valid_images.append(image_path) + valid_annotations.append(txt) + valid_indicies.append(i) + + if number_of_missing_images > 0: + print(f"*** WARNING *** missing {number_of_missing_images} files.") + + with codecs.open( + os.path.join(root, OUTPUT_FILENAME_TEMPLATE.format(lang_code)), + "w", + encoding="utf-8", + ) as fp: + json.dump( + { + "image_paths": valid_images, + "annotations": valid_annotations, + "indicies": valid_indicies, + }, + fp, + ensure_ascii=False, + ) diff --git a/perception_models/apps/pe/clip_benchmark/metrics/__captioning.py b/perception_models/apps/pe/clip_benchmark/metrics/__captioning.py new file mode 100644 index 0000000000000000000000000000000000000000..23c7db8494f9e940b0e2c376178a9c96303d08a6 --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/metrics/__captioning.py @@ -0,0 +1,119 @@ +import json + +from pycocoevalcap.bleu.bleu import Bleu +from pycocoevalcap.cider.cider import Cider +from pycocoevalcap.meteor.meteor import Meteor +from pycocoevalcap.rouge.rouge import Rouge +from pycocoevalcap.spice.spice import Spice +from pycocoevalcap.tokenizer.ptbtokenizer import PTBTokenizer +# from open_clip import tokenize +from tqdm.auto import tqdm + +# from open_clip.tokenizer import _tokenizer +from core.vision_encoder.tokenizer import _tokenizer, tokenize + +""" +Code adapted from https://github.com/salaniz/pycocoevalcap/blob/master/eval.py +Thanks to @salaniz for the code! +""" + + +class COCOEvalCap: + def __init__(self, results): + self.evalImgs = [] + self.eval = {} + self.imgToEval = {} + self.results = results + + def evaluate(self): + gts = {} + res = {} + for imgId, r in enumerate(self.results): + gts[imgId] = r["true"] + res[imgId] = r["gen"] + # ================================================= + # Set up scorers + # ================================================= + print("tokenization...") + tokenizer = PTBTokenizer() + gts = tokenizer.tokenize(gts) + res = tokenizer.tokenize(res) + + # ================================================= + # Set up scorers + # ================================================= + print("setting up scorers...") + scorers = [ + (Bleu(4), ["Bleu_1", "Bleu_2", "Bleu_3", "Bleu_4"]), + (Meteor(), "METEOR"), + (Rouge(), "ROUGE_L"), + (Cider(), "CIDEr"), + (Spice(), "SPICE"), + ] + + # ================================================= + # Compute scores + # ================================================= + for scorer, method in scorers: + print("computing %s score..." % (scorer.method())) + score, scores = scorer.compute_score(gts, res) + if type(method) == list: + for sc, scs, m in zip(score, scores, method): + self.setEval(sc, m) + self.setImgToEvalImgs(scs, gts.keys(), m) + print("%s: %0.3f" % (m, sc)) + else: + self.setEval(score, method) + self.setImgToEvalImgs(scores, gts.keys(), method) + print("%s: %0.3f" % (method, score)) + self.setEvalImgs() + + def setEval(self, score, method): + self.eval[method] = score + + def setImgToEvalImgs(self, scores, imgIds, method): + for imgId, score in zip(imgIds, scores): + if not imgId in self.imgToEval: + self.imgToEval[imgId] = {} + self.imgToEval[imgId]["image_id"] = imgId + self.imgToEval[imgId][method] = score + + def setEvalImgs(self): + self.evalImgs = [eval for imgId, eval in self.imgToEval.items()] + + +def evaluate( + model, + dataloader, + batch_size, + device, + transform, + train_dataloader=None, + num_workers=None, + amp=True, + verbose=False, +): + results = [] + image_id = 0 + gt = [] + for idx, (img, captions) in enumerate(tqdm(dataloader)): + out = model.generate(img.to(device)) + decoded = [ + _tokenizer.decode(i) + .split("")[0] + .replace("", "") + .strip() + for i in out.cpu().numpy() + ] + for pred, true in zip(decoded, captions): + true = [{"caption": t} for t in true] + pred = [{"caption": pred}] + results.append({"image_id": image_id, "gen": pred, "true": true}) + image_id += 1 + coco_eval = COCOEvalCap(results) + coco_eval.evaluate() + metrics = coco_eval.eval + # print output evaluation scores + for metric, score in metrics.items(): + print(f"{metric}: {score:.3f}") + return metrics diff --git a/perception_models/apps/pe/clip_benchmark/metrics/__init__.py b/perception_models/apps/pe/clip_benchmark/metrics/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/perception_models/apps/pe/clip_benchmark/metrics/image_caption_selection.py b/perception_models/apps/pe/clip_benchmark/metrics/image_caption_selection.py new file mode 100644 index 0000000000000000000000000000000000000000..e72b333c75dc4a06b843b44f1c783fd4742f992a --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/metrics/image_caption_selection.py @@ -0,0 +1,93 @@ +import logging +from contextlib import suppress + +import torch +import torch.nn.functional as F +from open_clip import image_to_device +from tqdm import tqdm + + +def evaluate(model, dataloader, tokenizer, device, amp=True, args=None): + """ + Evaluate the model on the given dataset. + The task has N instances, each instance has I images and C captions. + For each instance, the goal is to find the correct image for each caption and the correct caption for each image. + This is done by computing the similarities between each image and each caption. + This procedure is used to evaluate the models on Winoground and SugarCrepe. + + Parameters + ---------- + + model: torch.nn,Module + CLIP-like model with `encode_image` and `encode_text` + + dataloader: torch.utils.data.Dataloader + dataloader to use for evaluation + + tokenizer: + text tokenizer, i.e. convert list of strings to torch.Tensor of integers + + device: cpu/cuda + + amp: whether to use automatic mixed precision + + Returns + ------- + + dict of accuracy metrics + """ + autocast = torch.cuda.amp.autocast if amp else suppress + image_score = [] + text_score = [] + score = [] + for batch_images, batch_texts in tqdm(dataloader): + # assert(len(batch_images.shape) == 4) + batch_images = image_to_device( + batch_images, + device, + torch.float32, + mean=args.image_mean, + std=args.image_std, + ) + # Because of the packing collate function we cannot support multi-image to caption selection + nim = 1 + + # tokenize all texts in the batch + nt = len(batch_texts[0]) + batch_texts_tok_ = tokenizer( + [text for i, texts in enumerate(batch_texts) for text in texts] + ).to(device) + + # compute the embedding of images and texts + with torch.no_grad(), autocast(): + batch_images_emb = F.normalize( + model.encode_image(batch_images), dim=-1 + ).unsqueeze(1) + B, _, emb_dim = batch_images_emb.shape + batch_texts_emb = F.normalize( + model.encode_text(batch_texts_tok_), dim=-1 + ).view(B, nt, -1) + + gt = torch.arange(min(nim, nt)).to(device) + for i in range(B): + # iteratve over instances + + # compute similarities between each image and each text + images_emb = batch_images_emb[i] + texts_emb = batch_texts_emb[i] + scores = images_emb @ texts_emb.t() + + # i-th image should be matched to the i-th text + image_closest_text = scores.argmax(dim=1)[: len(gt)] + text_closest_image = scores.argmax(dim=0)[: len(gt)] + pred_text_is_correct = (image_closest_text == gt).all().item() + pred_image_is_correct = (text_closest_image == gt).all().item() + all_correct = pred_text_is_correct and pred_image_is_correct + image_score.append(pred_image_is_correct) + text_score.append(pred_text_is_correct) + score.append(all_correct) + metrics = {} + metrics["image_acc"] = torch.Tensor(image_score).float().mean().item() + metrics["text_acc"] = torch.Tensor(text_score).float().mean().item() + metrics["acc"] = torch.Tensor(score).float().mean().item() + return metrics diff --git a/perception_models/apps/pe/clip_benchmark/metrics/linear_probe.py b/perception_models/apps/pe/clip_benchmark/metrics/linear_probe.py new file mode 100644 index 0000000000000000000000000000000000000000..2ba38a365bbfc07e5a75b1289bfc7569a77279ba --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/metrics/linear_probe.py @@ -0,0 +1,453 @@ +import os +import time +from contextlib import suppress + +import numpy as np +import torch +import torch.nn.functional as F +from sklearn.metrics import balanced_accuracy_score, classification_report +from torch.utils.data import DataLoader, Dataset, Sampler +from tqdm import tqdm + +from .zeroshot_classification import accuracy + + +def assign_learning_rate(param_group, new_lr): + param_group["lr"] = new_lr + + +def _warmup_lr(base_lr, warmup_length, step): + return base_lr * (step + 1) / warmup_length + + +def cosine_lr(optimizer, base_lrs, warmup_length, steps): + if not isinstance(base_lrs, list): + base_lrs = [base_lrs for _ in optimizer.param_groups] + assert len(base_lrs) == len(optimizer.param_groups) + + def _lr_adjuster(step): + for param_group, base_lr in zip(optimizer.param_groups, base_lrs): + if step < warmup_length: + lr = _warmup_lr(base_lr, warmup_length, step) + else: + e = step - warmup_length + es = steps - warmup_length + lr = 0.5 * (1 + np.cos(np.pi * e / es)) * base_lr + assign_learning_rate(param_group, lr) + + return _lr_adjuster + + +class Featurizer(torch.nn.Module): + def __init__(self, model, normalize=True): + super().__init__() + self.model = model + self.normalize = normalize + + def forward(self, input): + image_features = self.model.encode_image(input) + if self.normalize: + image_features = F.normalize(image_features, dim=-1) + return image_features + + +class FeatureDataset(Dataset): + def __init__(self, features, targets): + self.features = features + self.targets = targets + + def __len__(self): + return len(self.features) + + def __getitem__(self, i): + return self.features[i], self.targets[i] + + +def train( + dataloader, + input_shape, + output_shape, + weight_decay, + lr, + epochs, + autocast, + device, + seed, +): + torch.manual_seed(seed) + model = torch.nn.Linear(input_shape, output_shape) + devices = [x for x in range(torch.cuda.device_count())] + model = model.cuda() + model = torch.nn.DataParallel(model, device_ids=devices) + optimizer = torch.optim.AdamW( + model.parameters(), + lr=lr, + weight_decay=weight_decay, + ) + criterion = torch.nn.CrossEntropyLoss() + + len_loader = len(dataloader) + scheduler = cosine_lr(optimizer, lr, 0.0, epochs * len_loader) + + for epoch in range(epochs): + end = time.time() + for i, (x, y) in enumerate(dataloader): + x, y = x.cuda(), y.cuda() + step = i + epoch * len_loader + data_time = time.time() - end + scheduler(step) + + optimizer.zero_grad() + with autocast(): + pred = model(x) + loss = criterion(pred, y) + + loss.backward() + optimizer.step() + + batch_time = time.time() - end + end = time.time() + + if (i % 20) == 1: + num_samples = i * len(x) + try: + samples_per_epoch = len(dataloader) + percent_complete = 100.0 * i / len(dataloader) + progress_message = ( + f"[{num_samples}/{samples_per_epoch} ({percent_complete:.0f}%)]" + ) + except TypeError: + progress_message = f"[{num_samples} samples]" + print( + f"Train Epoch: {epoch} {progress_message}\t" + f"Loss: {loss.item():.6f}\tData (t) {data_time:.3f}\tBatch (t) {batch_time:.3f}\t" + f"LR {optimizer.param_groups[0]['lr']:.5f}" + ) + return model + + +def infer(model, dataloader, autocast, device): + true, pred = [], [] + with torch.no_grad(): + for x, y in tqdm(dataloader): + x = x.to(device) + y = y.to(device) + + with autocast(): + logits = model(x) + + pred.append(logits.cpu()) + true.append(y.cpu()) + + logits = torch.cat(pred) + target = torch.cat(true) + return logits, target + + +def find_peak( + wd_list, + idxs, + train_loader, + val_loader, + input_shape, + output_shape, + lr, + epochs, + autocast, + device, + verbose, + seed, +): + best_wd_idx, max_acc = 0, 0 + for idx in idxs: + weight_decay = wd_list[idx] + model = train( + train_loader, + input_shape, + output_shape, + weight_decay, + lr, + epochs, + autocast, + device, + seed, + ) + logits, target = infer(model, val_loader, autocast, device) + (acc1,) = accuracy(logits.float(), target.float(), topk=(1,)) + if verbose: + print(f"Valid accuracy with weight_decay {weight_decay}: {acc1}") + if max_acc < acc1: + best_wd_idx, max_acc = idx, acc1 + return best_wd_idx + + +def evaluate( + model, + train_dataloader, + dataloader, + fewshot_k, + batch_size, + num_workers, + lr, + epochs, + model_id, + seed, + feature_root, + device, + val_dataloader=None, + normalize=True, + amp=True, + verbose=False, +): + assert device == "cuda" # need to use cuda for this else too slow + # first we need to featurize the dataset, and store the result in feature_root + if not os.path.exists(feature_root): + os.mkdir(feature_root) + feature_dir = os.path.join(feature_root, model_id) + if not os.path.exists(feature_dir): + os.mkdir(feature_dir) + + featurizer = Featurizer(model, normalize).cuda() + autocast = torch.cuda.amp.autocast if amp else suppress + if not os.path.exists(os.path.join(feature_dir, "targets_train.pt")): + # now we have to cache the features + devices = [x for x in range(torch.cuda.device_count())] + featurizer = torch.nn.DataParallel(featurizer, device_ids=devices) + + splits = ["_train", "_val", "_test"] + for save_str, loader in zip( + splits, [train_dataloader, val_dataloader, dataloader] + ): + if loader is None: + continue + features = [] + targets = [] + num_batches_tracked = 0 + num_cached = 0 + with torch.no_grad(): + for images, target in tqdm(loader): + images = images.to(device) + + with autocast(): + feature = featurizer(images) + + features.append(feature.cpu()) + targets.append(target) + + num_batches_tracked += 1 + if (num_batches_tracked % 100) == 0: + features = torch.cat(features) + targets = torch.cat(targets) + + torch.save( + features, + os.path.join( + feature_dir, f"features{save_str}_cache_{num_cached}.pt" + ), + ) + torch.save( + targets, + os.path.join( + feature_dir, f"targets{save_str}_cache_{num_cached}.pt" + ), + ) + num_cached += 1 + features = [] + targets = [] + + if len(features) > 0: + features = torch.cat(features) + targets = torch.cat(targets) + torch.save( + features, + os.path.join( + feature_dir, f"features{save_str}_cache_{num_cached}.pt" + ), + ) + torch.save( + targets, + os.path.join( + feature_dir, f"targets{save_str}_cache_{num_cached}.pt" + ), + ) + num_cached += 1 + + features = torch.load( + os.path.join(feature_dir, f"features{save_str}_cache_0.pt") + ) + targets = torch.load( + os.path.join(feature_dir, f"targets{save_str}_cache_0.pt") + ) + for k in range(1, num_cached): + next_features = torch.load( + os.path.join(feature_dir, f"features{save_str}_cache_{k}.pt") + ) + next_targets = torch.load( + os.path.join(feature_dir, f"targets{save_str}_cache_{k}.pt") + ) + features = torch.cat((features, next_features)) + targets = torch.cat((targets, next_targets)) + + for k in range(num_cached): + os.remove(os.path.join(feature_dir, f"features{save_str}_cache_{k}.pt")) + os.remove(os.path.join(feature_dir, f"targets{save_str}_cache_{k}.pt")) + + torch.save(features, os.path.join(feature_dir, f"features{save_str}.pt")) + torch.save(targets, os.path.join(feature_dir, f"targets{save_str}.pt")) + + features = torch.load(os.path.join(feature_dir, "features_train.pt")) + targets = torch.load(os.path.join(feature_dir, "targets_train.pt")) + + # second, make a dataloader with k features per class. if k = -1, use all features. + length = len(features) + perm = [p.item() for p in torch.randperm(length)] + idxs = [] + counts = {} + num_classes = 0 + + for p in perm: + target = targets[p].item() + if target not in counts: + counts[target] = 0 + num_classes += 1 + + if fewshot_k < 0 or counts[target] < fewshot_k: + counts[target] += 1 + idxs.append(p) + + for c in counts: + if fewshot_k > 0 and counts[c] != fewshot_k: + print("insufficient data for this eval") + return + + train_features = features[idxs] + train_labels = targets[idxs] + if val_dataloader is not None: + features_val = torch.load(os.path.join(feature_dir, "features_val.pt")) + targets_val = torch.load(os.path.join(feature_dir, "targets_val.pt")) + feature_val_dset = FeatureDataset(features_val, targets_val) + feature_val_loader = DataLoader( + feature_val_dset, + batch_size=batch_size, + shuffle=True, + num_workers=num_workers, + pin_memory=True, + ) + feature_train_val_dset = FeatureDataset( + np.concatenate((train_features, features_val)), + np.concatenate((train_labels, targets_val)), + ) + feature_train_val_loader = DataLoader( + feature_train_val_dset, + batch_size=batch_size, + shuffle=True, + num_workers=num_workers, + pin_memory=True, + ) + feature_train_dset = FeatureDataset(train_features, train_labels) + feature_train_loader = DataLoader( + feature_train_dset, + batch_size=batch_size, + shuffle=True, + num_workers=num_workers, + pin_memory=True, + ) + features_test = torch.load(os.path.join(feature_dir, "features_test.pt")) + targets_test = torch.load(os.path.join(feature_dir, "targets_test.pt")) + feature_test_dset = FeatureDataset(features_test, targets_test) + feature_test_loader = DataLoader( + feature_test_dset, + batch_size=batch_size, + shuffle=True, + num_workers=num_workers, + pin_memory=True, + ) + input_shape, output_shape = features[0].shape[0], targets.max().item() + 1 + if val_dataloader is not None: + # perform openAI-like hyperparameter sweep + # https://arxiv.org/pdf/2103.00020.pdf A.3 + # instead of scikit-learn LBFGS use FCNNs with AdamW + wd_list = np.logspace(-6, 2, num=97).tolist() + wd_list_init = np.logspace(-6, 2, num=7).tolist() + wd_init_idx = [i for i, val in enumerate(wd_list) if val in wd_list_init] + peak_idx = find_peak( + wd_list, + wd_init_idx, + feature_train_loader, + feature_val_loader, + input_shape, + output_shape, + lr, + epochs, + autocast, + device, + verbose, + seed, + ) + step_span = 8 + while step_span > 0: + left, right = max(peak_idx - step_span, 0), min( + peak_idx + step_span, len(wd_list) - 1 + ) + peak_idx = find_peak( + wd_list, + [left, peak_idx, right], + feature_train_loader, + feature_val_loader, + input_shape, + output_shape, + lr, + epochs, + autocast, + device, + verbose, + seed, + ) + step_span //= 2 + best_wd = wd_list[peak_idx] + train_loader = feature_train_val_loader + else: + best_wd = 0 + train_loader = feature_train_loader + + final_model = train( + train_loader, + input_shape, + output_shape, + best_wd, + lr, + epochs, + autocast, + device, + seed, + ) + logits, target = infer(final_model, feature_test_loader, autocast, device) + pred = logits.argmax(axis=1) + + # measure accuracy + if target.max() >= 5: + acc1, acc5 = accuracy(logits.float(), target.float(), topk=(1, 5)) + else: + (acc1,) = accuracy(logits.float(), target.float(), topk=(1,)) + acc5 = float("nan") + mean_per_class_recall = balanced_accuracy_score(target, pred) + fair_info = { + "weight_decay": best_wd, + "acc1": acc1, + "acc5": acc5, + "mean_per_class_recall": mean_per_class_recall, + "classification_report": classification_report(target, pred, digits=3), + } + if verbose: + print(fair_info["classification_report"]) + print(f"Test acc1: {acc1} with weight_decay: {best_wd}") + return { + "lp_acc1": fair_info["acc1"], + "lp_acc5": fair_info["acc5"], + "lp_mean_per_class_recall": fair_info["mean_per_class_recall"], + "weight_decay": fair_info["weight_decay"], + "epochs": epochs, + "seed": seed, + "fewshot_k": fewshot_k, + "normalized": normalize, + } diff --git a/perception_models/apps/pe/clip_benchmark/metrics/multiclass_retrieval.py b/perception_models/apps/pe/clip_benchmark/metrics/multiclass_retrieval.py new file mode 100644 index 0000000000000000000000000000000000000000..057d48301b4297919f7ff8064b10d187dffb9b54 --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/metrics/multiclass_retrieval.py @@ -0,0 +1,129 @@ +import json +import logging +from contextlib import suppress + +import numpy as np +import torch +import torch.nn.functional as F +from clip_benchmark.metrics.zeroshot_retrieval import (dataloader_with_indices, + recall_at_k) +from tqdm import tqdm + + +def evaluate( + model, + dataloader, + tokenizer, + device, + amp=True, + recall_k_list=[1], + args=None, + retrieval_template=None, +): + """ + Evaluate the model on the given dataset + + Parameters + ---------- + + model: torch.nn,Module + CLIP-like model with `encode_image` and `encode_text` + + dataloader: torch.utils.data.Dataloader + dataloader to use for evaluation + + tokenizer: + text tokenizer, i.e. convert list of strings to torch.Tensor of integers + + device: cpu/cuda + + amp: whether to use automatic mixed precision + + recall_k_list: list of int + recall@k k's to use + + retrieval_template: + dict of retrieval templates for each class. Retrieval templates should contain lists of image/text indexes. The model will performed retrieval accross the examples in each list. + + Returns + ------- + + dict of retrieval metrics + """ + # list of batch of images embedding + batch_images_emb_list = [] + # list of batch of text embedding + batch_texts_emb_list = [] + # for each text, we collect the corresponding image index, as each image can have multiple corresponding texts + texts_image_index = [] + dataloader = dataloader_with_indices(dataloader) + autocast = torch.cuda.amp.autocast if amp else suppress + + for batch_images, batch_texts, inds in tqdm(dataloader): + # move the batch to the device + batch_images = image_to_device( + batch_images, + device, + torch.float32, + mean=args.image_mean, + std=args.image_std, + ) + + # tokenize all texts in the batch + batch_texts_tok = tokenizer( + [text for i, texts in enumerate(batch_texts) for text in texts] + ).to(device) + + # compute the embedding of images and texts + with torch.no_grad(), autocast(): + batch_images_emb = F.normalize(model.encode_image(batch_images), dim=-1) + batch_texts_emb = F.normalize(model.encode_text(batch_texts_tok), dim=-1) + + batch_images_emb_list.append(batch_images_emb.cpu()) + batch_texts_emb_list.append(batch_texts_emb.cpu()) + + batch_size = len(batch_images_emb_list[0]) + + # concatenate all embeddings + images_emb = torch.cat(batch_images_emb_list) + texts_emb = torch.cat(batch_texts_emb_list) + + assert images_emb.shape[0] == texts_emb.shape[0] + + # get the score for each text and image pair + scores = texts_emb @ images_emb.t() + + metrics = {} + multiclass_image_retrieval = [] + multiclass_text_retrieval = [] + for c in retrieval_template.keys(): + + image_retrieval = [] + text_retrieval = [] + for indexes in retrieval_template[c]: + retrieved = scores[np.ix_(indexes, indexes)] + positive_pairs = torch.zeros_like(retrieved, dtype=bool) + positive_pairs[ + torch.arange(len(retrieved)), torch.arange(len(retrieved)) + ] = True + + image_retrieval.append(recall_at_k(retrieved, positive_pairs, k=1)) + text_retrieval.append(recall_at_k(retrieved.T, positive_pairs, k=1)) + + average_image_retrieval = torch.cat(image_retrieval).float().mean().item() + average_text_retrieval = torch.cat(text_retrieval).float().mean().item() + + metrics[f"image_retrieval_recall@1_{c}"] = average_image_retrieval + metrics[f"text_retrieval_recall@1_{c}"] = average_text_retrieval + + multiclass_image_retrieval.append(average_image_retrieval) + multiclass_text_retrieval.append(average_text_retrieval) + + metrics["image_retrieval_recall@1_multiclass"] = ( + torch.tensor(multiclass_image_retrieval).float().mean().item() + ) + metrics["text_retrieval_recall@1_multiclass"] = ( + torch.tensor(multiclass_text_retrieval).float().mean().item() + ) + + return metrics diff --git a/perception_models/apps/pe/clip_benchmark/metrics/visualization.py b/perception_models/apps/pe/clip_benchmark/metrics/visualization.py new file mode 100644 index 0000000000000000000000000000000000000000..994006c0fb833317a3bc8aae7e7500ca9f93b55a --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/metrics/visualization.py @@ -0,0 +1,150 @@ +import io +import logging +import pickle +from contextlib import suppress + +import numpy as np +import torch +import torch.nn as nn +import torch.nn.functional as F +from sklearn.metrics import balanced_accuracy_score, classification_report +from tqdm import tqdm + +# from open_clip import image_to_device + + +def evaluate(model, dataloader, device, visargs, amp=True, args=None): + autocast = torch.cuda.amp.autocast if amp else suppress + model.visual.output_tokens = True + + if visargs.delete_post_ln: + model.visual.ln_post = nn.Identity() + if visargs.extract_layer is not None and visargs.extract_layer != -1: + model.visual.transformer.resblocks = model.visual.transformer.resblocks[ + : (visargs.extract_layer + 1) + ] + + if visargs.attn_rollout is not None: + for module in model.visual.transformer.resblocks: + if hasattr(module, "attn"): + module.attn.forward = module.attn.forward_with_attn + + with torch.no_grad(): + for images, target in dataloader: + old_images = images + # images = image_to_device(images, device, torch.float32, mean=args.image_mean, std=args.image_std) + iamges = images.to(device) + target = target.to(device) + + with autocast(): + # predict + if hasattr(model.visual, "trunk"): + x = model.visual.trunk.forward_features(images) + else: + x = model.visual(images).latent + + # apply attn pooler if we want + if visargs.attn_pooling is not None: + idx = {"Q": 0, "K": 1, "V": 2}[visargs.attn_pooling] + attn_layer = model.visual.attn_pool.attn + mat = attn_layer.in_proj_weight[ + attn_layer.embed_dim * idx : attn_layer.embed_dim * (idx + 1) + ] + x = x @ mat.T + + if attn_layer.in_proj_bias is not None: + x = ( + x + + attn_layer.in_proj_bias[ + None, + None, + idx + * attn_layer.embed_dim : (idx + 1) + * attn_layer.embed_dim, + ] + ) + + x = x.reshape(target.shape[0], -1, x.shape[-1]) + + if isinstance(images, list): + old_images = old_images[0] + + if ( + hasattr(model.visual, "embed_cls_token") + and model.visual.embed_cls_token + ) or ( + hasattr(model.visual, "trunk") + and hasattr(model.visual.trunk, "cls_token") + and model.visual.trunk.cls_token + ): + if isinstance(images, list): + x = x[:, :-1] + old_images = old_images.reshape( + target.shape[0], -1, *old_images.shape[1:] + ) + old_images = old_images[:, :-1] + else: + x = x[:, 1:] + + sidelen = int(x.shape[1] ** 0.5) + + if isinstance(images, list): + old_images = old_images.reshape( + target.shape[0], sidelen, sidelen, 3, *old_images.shape[-2:] + ) + psz = old_images.shape[-1] + old_images = old_images.permute(0, 3, 1, 4, 2, 5).reshape( + target.shape[0], 3, sidelen * psz, sidelen * psz + ) + + x = x.view(x.shape[0], sidelen, sidelen, -1) + x = x.permute(0, 3, 1, 2).contiguous() + # convert on umpy + x = x.cpu().numpy().astype(np.float16) + + old_images = old_images.cpu().numpy().astype(np.float16) + old_images *= np.array(args.image_std)[None, :, None, None] + old_images += np.array(args.image_mean)[None, :, None, None] + data = {"image": old_images, "features": x} + + if visargs.attn_pooling is not None: + data["attn"] = { + "qkv": attn_layer.in_proj_weight.data, + "bias": ( + attn_layer.in_proj_bias.data + if attn_layer.in_proj_bias is not None + else None + ), + "probe": model.visual.attn_pool.probe, + } + + if visargs.attn_rollout: + attns = [ + module.attn.attn.mean(dim=1) + for module in model.visual.transformer.resblocks + if hasattr(module, "attn") + ] + + def roll(i, rollout=None): + for attn in attns[i:]: + rollout = ( + attn.float() @ rollout + if rollout is not None + else attn.float() + ) + return rollout + + # rollout = torch.stack([roll(i) for i in range(len(attns))], dim=1) + rollout = torch.stack(attns, dim=1) + data["rollout"] = rollout.cpu().numpy().astype(np.float16) + + bytes_str = pickle.dumps(data) + + print(f"<<<<<<<<<<<<<") + print(bytes_str) + print(f">>>>>>>>>>>>>") + + break + + exit() + return None diff --git a/perception_models/apps/pe/clip_benchmark/metrics/zeroshot_classification.py b/perception_models/apps/pe/clip_benchmark/metrics/zeroshot_classification.py new file mode 100644 index 0000000000000000000000000000000000000000..ec0ceb5728d3c15fef92a5b89c26b2505dd481bd --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/metrics/zeroshot_classification.py @@ -0,0 +1,295 @@ +""" +Code adapated from https://github.com/mlfoundations/open_clip/blob/main/src/training/zero_shot.py +Thanks to the authors of OpenCLIP +""" + +import logging +from contextlib import suppress + +import torch +import torch.nn.functional as F +from sklearn.metrics import balanced_accuracy_score, classification_report +from tqdm import tqdm + +# from open_clip import image_to_device + + +def zero_shot_classifier(model, tokenizer, classnames, templates, device, amp=True): + """ + This function returns zero-shot vectors for each class in order + to use it for zero-shot classification. + + + model: + CLIP-like model with `encode_text` + + tokenizer: + text tokenizer, i.e. convert list of strings to torch.Tensor of integers + + classnames: list of str + name of classes + + templates: list of str + templates to use. + + Returns + ------- + + torch.Tensor of shape (N,C) where N is the number + of templates, and C is the number of classes. + """ + import os + + if os.environ.get("CLIP_BENCHMARK_DEBUG", None) == "1": + breakpoint() + + autocast = torch.cuda.amp.autocast if amp else suppress + with torch.no_grad(), autocast(): + zeroshot_weights = [] + for classname in tqdm(classnames): + if type(templates) == dict: + # class-specific prompts (e.g., CuPL https://arxiv.org/abs/2209.03320) + texts = templates[classname] + elif type(templates) == list: + # generic prompts tht are specialized for each class by replacing {c} with the class name + texts = [template.format(c=classname) for template in templates] + else: + raise ValueError("templates must be a list or a dict") + texts = tokenizer(texts).to(device) # tokenize + class_embeddings = model.encode_text(texts) + class_embedding = F.normalize(class_embeddings, dim=-1).mean(dim=0) + class_embedding /= class_embedding.norm() + zeroshot_weights.append(class_embedding) + zeroshot_weights = torch.stack(zeroshot_weights, dim=1).to(device) + return zeroshot_weights + + +def accuracy(output, target, topk=(1,)): + """ + Compute top-k accuracy + + output: torch.Tensor + shape (N, C) where N is the number of examples, C the number of classes. + these are the logits. + + target: torch.Tensor + shape (N,) where N is the number of examples. Groundtruth class id of each example. + + topk: tuple + which topk to compute, e.g., topk=(1,5) will compute top-1 and top-5 accuracies + + Returns + ------- + + list of top-k accuracies in the same order as `topk` + """ + pred = output.topk(max(topk), 1, True, True)[1].t() + correct = pred.eq(target.view(1, -1).expand_as(pred)) + n = len(target) + return [ + float(correct[:k].reshape(-1).float().sum(0, keepdim=True).cpu().numpy()) / n + for k in topk + ] + + +def run_classification( + model, classifier, dataloader, device, video_dataset=False, amp=True, args=None +): + """ + Run zero-shot classifcation + + model: torch.nn.Module + CLIP-like model with `encode_image` and `encode_text` + + classifier: torch.Tensor + obtained from the function `zero_shot_classifier` + + dataloader: torch.utils.data.Dataloader + + Returns + ------- + (pred, true) where + - pred (N, C) are the logits + - true (N,) are the actual classes + """ + autocast = torch.cuda.amp.autocast if amp else suppress + pred = [] + true = [] + nb = 0 + with torch.no_grad(): + import os + + if os.environ.get("CLIP_BENCHMARK_DEBUG", None) == "1": + breakpoint() + + for images, target in tqdm(dataloader): + + if isinstance(images, torch.Tensor): + images = images.to(device, torch.float32) + elif isinstance(images, (list, tuple)): # video frames as a list/tuple + images = [x.to(device, torch.float32) for x in images] + images = torch.stack(images, dim=0).permute(1,0,2,3,4).contiguous() # nbchw -> bncwh + else: + raise NotImplementedError + + target = target.to(device) + + with autocast(): + # predict + if video_dataset: + image_features = model.encode_video(images) + else: + image_features = model.encode_image(images) + + image_features = F.normalize(image_features, dim=-1) + logits = 100.0 * image_features @ classifier + + true.append(target.cpu()) + pred.append(logits.float().cpu()) + + pred = torch.cat(pred) + true = torch.cat(true) + return pred, true + + +def average_precision_per_class(scores, targets): + """ + Compute average precision for each class + this metric is used for multi-label classification + see explanations here https://fangdahan.medium.com/calculate-mean-average-precision-map-for-multi-label-classification-b082679d31be + Code is adapted from https://github.com/pytorch/tnt/blob/master/torchnet/meter/meter.py, thanks to the authors of `tnt`. + + Parameters + ---------- + + scores: torch.Tensor + logits, of shape (N,C) where N is the number of examples, C the number of classes + + targets: torch.Tensor + one-hot vectors of groundtruth targets (N, C), where N is the number of examples, C is the + number of classes + + Returns + ------- + + torch.Tensor of shape (C,) of avereage precision for each class, where C is + the number of classes. + + """ + ap = torch.zeros(scores.size(1)) + rg = torch.arange(1, scores.size(0) + 1).float() + # compute average precision for each class + for k in range(scores.size(1)): + # sort scores + scores_k = scores[:, k] + targets_k = targets[:, k] + _, sortind = torch.sort(scores_k, 0, True) + truth = targets_k[sortind] + tp = truth.float().cumsum(0) + # compute precision curve + precision = tp.div(rg) + # compute average precision + ap[k] = precision[truth.bool()].sum() / max(float(truth.sum()), 1) + return ap + + +def evaluate( + model, + dataloader, + tokenizer, + classnames, + templates, + device, + video_dataset=False, + amp=True, + verbose=False, + save_clf=None, + load_clfs=[], + args=None, +): + """ + Run zero-shot classification and evaluate the metrics + + Parameters + ---------- + + model: torch.nn.Module + CLIP-like model with `encode_image` and `encode_text` + + dataloader: torch.utils.data.Dataloader + + tokenizer: text tokenizer + + classnames: list of str + class names + + templates: list of str + templates to use for zero-shot classification + + device: cpu/cuda + + amp: whether to use automatic mixed precision + + verbose: whether to use verbose model + + Returns + ------- + + dict of classification metrics + """ + if len(load_clfs) > 0: + n = len(load_clfs) + classifier = torch.load(load_clfs[0], map_location="cpu") / n + for i in range(1, n): + classifier = classifier + torch.load(load_clfs[i], map_location="cpu") / n + classifier = classifier.to(device) + else: + classifier = zero_shot_classifier( + model, tokenizer, classnames, templates, device, amp=amp + ) + + if save_clf is not None: + torch.save(classifier, save_clf) + # exit() - not sure if we want to exit here or not. + + logits, target = run_classification( + model, + classifier, + dataloader, + device, + video_dataset=video_dataset, + amp=amp, + args=args, + ) + is_multilabel = len(target.shape) == 2 + + if is_multilabel: + if verbose: + print("Detected a multi-label classification dataset") + # Multiple labels per image, multiple classes on the dataset + ap_per_class = average_precision_per_class(logits, target) + if verbose: + for class_name, ap in zip( + dataloader.dataset.classes, ap_per_class.tolist() + ): + print(f"Class: {class_name}, AveragePrecision: {ap}") + return {"mean_average_precision": ap_per_class.mean().item()} + else: + # Single label per image, multiple classes on the dataset + # just compute accuracy and mean_per_class_recall + + pred = logits.argmax(axis=1) + # measure accuracy + if len(dataloader.dataset.classes) >= 5: + acc1, acc5 = accuracy(logits, target, topk=(1, 5)) + else: + (acc1,) = accuracy(logits, target, topk=(1,)) + acc5 = float("nan") + mean_per_class_recall = balanced_accuracy_score(target, pred) + if verbose: + print(classification_report(target, pred, digits=3)) + return { + "acc1": acc1, + "acc5": acc5, + "mean_per_class_recall": mean_per_class_recall, + } diff --git a/perception_models/apps/pe/clip_benchmark/metrics/zeroshot_retrieval.py b/perception_models/apps/pe/clip_benchmark/metrics/zeroshot_retrieval.py new file mode 100644 index 0000000000000000000000000000000000000000..f475b094fcc63b1d8d70a435ff7ab8ea4a87c3ff --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/metrics/zeroshot_retrieval.py @@ -0,0 +1,252 @@ +from contextlib import suppress + +import torch +import torch.nn.functional as F +from tqdm import tqdm +from transformers import BatchFeature +from core.audio_visual_encoder import PEAudioVisual +from collections import defaultdict + + +def get_pe_av_embeddings(inputs, model, transform, this_modality, other_modality, device): + if this_modality == "text": + model_inputs = transform(text=inputs).to(device) + if other_modality == "video": + return model.encode_video_text(**model_inputs) + elif other_modality == "audio": + return model.encode_audio_text(**model_inputs) + else: + assert other_modality == "audio_video" + return model.encode_audio_video_text(**model_inputs) + elif this_modality == "audio": + model_inputs = transform(audio=inputs).to(device) + return model.encode_audio(**model_inputs) + elif this_modality == "video": + model_inputs = transform(videos=inputs).to(device) + return model.encode_video(**model_inputs) + else: + assert this_modality == "audio_video" + audios, videos = zip(*inputs) + model_inputs = transform(audio=audios, videos=videos).to(device) + return model.encode_audio_video(**model_inputs) + + +def evaluate( + model, + dataloader, + tokenizer, + device, + video_dataset=False, + amp=True, + recall_k_list=[1, 5, 10], + args=None, + audio_dataset=False, + transform=None, +): + """ + Evaluate the model on the given dataset + + Parameters + ---------- + + model: torch.nn,Module + CLIP-like model with `encode_(image|video|audio)` and `encode_text` + + dataloader: torch.utils.data.Dataloader + dataloader to use for evaluation + + tokenizer: + text tokenizer, i.e. convert list of strings to torch.Tensor of integers + + device: cpu/cuda + + amp: whether to use automatic mixed precision + + recall_k_list: list of int + recall@k k's to use + + Returns + ------- + + dict of retrieval metrics + """ + # list of batch of modality1 embedding + batch_modality1_emb_list = [] + # list of batch of modality2 embedding + batch_modality2_emb_list = [] + # for each text, we collect the corresponding media index, as each media can have multiple corresponding texts + modality2_media_index = [] + dataloader_wrapper = dataloader_with_indices(dataloader) + autocast = torch.cuda.amp.autocast if amp else suppress + + modality1 = "audio" if audio_dataset else "video" if video_dataset else "image" + modality2 = "video" if audio_dataset and video_dataset else "text" + + all_modality2 = [] + + for batch_modality1, batch_modality2, inds in tqdm(dataloader_wrapper): + all_modality2.extend(batch_modality2) + # store the index of media for each text + if isinstance(inds, torch.Tensor): + inds = inds.tolist() + batch_modality2_media_index = [ + ind for ind, texts in zip(inds, batch_modality2) for text in texts + ] + # compute the embedding + with torch.no_grad(), autocast(): + if isinstance(model, PEAudioVisual): + batch_modality1_emb = get_pe_av_embeddings( + inputs=batch_modality1, + model=model, + transform=transform, + this_modality=modality1, + other_modality=modality2, + device=device, + ) + batch_modality2_emb = get_pe_av_embeddings( + inputs=[i for batch in batch_modality2 for i in batch], + model=model, + transform=transform, + this_modality=modality2, + other_modality=modality1, + device=device, + ) + else: + # move the batch to the device + if isinstance(batch_modality1, torch.Tensor): + batch_modality1 = batch_modality1.to(device, torch.float32) + elif isinstance(batch_modality1, (list, tuple)): # video frames as a list/tuple + batch_modality1 = [x.to(device, torch.float32) for x in batch_modality1] + batch_modality1 = torch.stack(batch_modality1, dim=0).permute(1,0,2,3,4).contiguous() # nbchw -> bncwh + else: + raise NotImplementedError + + if video_dataset: + batch_modality1_emb = model.encode_video(batch_modality1, normalize=True) + else: + batch_modality1_emb = model.encode_image(batch_modality1, normalize=True) + tokenized = tokenizer( + [text for i, texts in enumerate(batch_modality2) for text in texts] + ).to(device) + batch_modality2_emb = model.encode_text(tokenized, normalize=True) + + batch_modality1_emb_list.append(batch_modality1_emb.cpu()) + batch_modality2_emb_list.append(batch_modality2_emb.cpu()) + modality2_media_index.extend(batch_modality2_media_index) + + batch_size = len(batch_modality1_emb_list[0]) + + # concatenate all embeddings + media_emb = torch.cat(batch_modality1_emb_list) + texts_emb = torch.cat(batch_modality2_emb_list) + + # get the score for each text and media pair + scores = texts_emb @ media_emb.t() + + scores_T = scores.T + + if args.reweight_retrieval: + scores = scores * args.reweight_scale + scores_T = scores_T * args.reweight_scale + scores = scores * scores.softmax(dim=0) + scores_T = scores_T * scores_T.softmax(dim=0) + + # construct a the positive pair matrix, which tells whether each text-media pair is a positive or not + positive_pairs = torch.zeros_like(scores, dtype=bool) + positive_pairs[torch.arange(len(scores)), modality2_media_index] = True + + all_modality2 = [x for y in all_modality2 for x in y] + modality2_index_mapping = defaultdict(set) + [modality2_index_mapping[modality2].add(i) for i, modality2 in enumerate(all_modality2)] + for indices in modality2_index_mapping.values(): + if len(indices) > 1: + # We have duplicate entries in modality2, so set them all to have the same labels + index_list = list(indices) + positive_pairs[index_list] = positive_pairs[index_list].any(dim=0) + + metrics = {} + for recall_k in recall_k_list: + # Note that recall_at_k computes **actual** recall i.e. nb_true_positive/nb_positives, where the number + # of true positives, e.g. for text retrieval, is, for each media, the number of retrieved texts matching that media among the top-k. + # Also, the number of positives are the total number of texts matching the media in the dataset, as we have a set of captions + # for each media, that number will be greater than 1 for text retrieval. + # However, media/text retrieval recall@k, the way it is done in CLIP-like papers, is a bit different. + # recall@k, in CLIP-like papers, is, for each media, either 1 or 0. It is 1 if atleast one text matches the media among the top-k. + # so we can easily compute that using the actual recall, by checking whether there is at least one true positive, + # which would be the case if the recall is greater than 0. One we compute the recal for each media (or text), we average + # it over the dataset. + metrics[f"{modality1}_retrieval_recall@{recall_k}"] = ( + ( + batchify( + recall_at_k, scores, positive_pairs, batch_size, device, k=recall_k + ) + > 0 + ) + .float() + .mean() + .item() + ) + metrics[f"{modality2}_retrieval_recall@{recall_k}"] = ( + ( + batchify( + recall_at_k, + scores_T, + positive_pairs.T, + batch_size, + device, + k=recall_k, + ) + > 0 + ) + .float() + .mean() + .item() + ) + + return metrics + + +def dataloader_with_indices(dataloader): + start = 0 + for x, y in dataloader: + end = start + len(y) + inds = torch.arange(start, end) + yield x, y, inds + start = end + + +def recall_at_k(scores, positive_pairs, k): + """ + Compute the recall at k for each sample + :param scores: compability score between text and media embeddings (nb texts, nb media) + :param k: number of media to consider per text, for retrieval + :param positive_pairs: boolean matrix of positive pairs (nb texts, nb media) + :return: recall at k averaged over all texts + """ + nb_texts, nb_media = scores.shape + # for each text, sort according to media scores in decreasing order + topk_indices = torch.topk(scores, k, dim=1)[1] + # compute number of positives for each text + nb_positive = positive_pairs.sum(dim=1) + # nb_texts, k, nb_media + topk_indices_onehot = torch.nn.functional.one_hot( + topk_indices, num_classes=nb_media + ) + # compute number of true positives + positive_pairs_reshaped = positive_pairs.view(nb_texts, 1, nb_media) + # a true positive means a positive among the topk + nb_true_positive = (topk_indices_onehot * positive_pairs_reshaped).sum(dim=(1, 2)) + # compute recall at k + recall_at_k = nb_true_positive / nb_positive + return recall_at_k + + +def batchify(func, X, Y, batch_size, device, *args, **kwargs): + results = [] + for start in range(0, len(X), batch_size): + end = start + batch_size + x = X[start:end].to(device) + y = Y[start:end].to(device) + result = func(x, y, *args, **kwargs).cpu() + results.append(result) + return torch.cat(results) diff --git a/perception_models/apps/pe/clip_benchmark/model_collection.py b/perception_models/apps/pe/clip_benchmark/model_collection.py new file mode 100644 index 0000000000000000000000000000000000000000..01e23e4fde4459704fd576e093d36f07880e8ed0 --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/model_collection.py @@ -0,0 +1,9 @@ +# import open_clip + + +def get_model_collection_from_file(path): + return [l.strip().split(",") for l in open(path).readlines()] + + +model_collection = { +} diff --git a/perception_models/apps/pe/clip_benchmark/tasks/wds_benchmarks.txt b/perception_models/apps/pe/clip_benchmark/tasks/wds_benchmarks.txt new file mode 100644 index 0000000000000000000000000000000000000000..abc2ffc9d94ac899b2f47445a787bfced89fa19d --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/tasks/wds_benchmarks.txt @@ -0,0 +1,16 @@ +# image classification +wds/wds_imagenet1k +wds/wds_imagenetv2 +wds/wds_imagenet-a +wds/wds_imagenet-r +wds/wds_imagenet_sketch + +# image retrieval +wds/wds_mscoco_captions +wds/wds_flickr30k + +# video classification +k400_val + +# video retrieval +msrvtt diff --git a/perception_models/apps/pe/clip_benchmark/webdataset_builder.py b/perception_models/apps/pe/clip_benchmark/webdataset_builder.py new file mode 100644 index 0000000000000000000000000000000000000000..40122b80d255a9bc5328cc2a936b18637e130808 --- /dev/null +++ b/perception_models/apps/pe/clip_benchmark/webdataset_builder.py @@ -0,0 +1,344 @@ +# Convert CLIP_benchmark datasets to webdataset format + +import argparse +import io +import os +import sys + +import torch +import torch.utils.data +import webdataset +from tqdm import tqdm + +from .datasets.builder import build_dataset + + +def get_parser_args(): + parser = argparse.ArgumentParser( + description=""" + Convert a CLIP_benchmark dataset to the webdataset format (TAR files). + Datasets can be uploaded to the Huggingface Hub to allow CLIP model + evaluation from anywhere with an Internet connection. + + To convert other image classification datasets, use the Python API: + >>> import clip_benchmark.webdataset_builder + >>> help(clip_benchmark.webdataset_builder.convert_dataset) + """ + ) + # Main arguments + parser.add_argument( + "--dataset", + "-d", + required=True, + type=str, + help="CLIP_benchmark compatible dataset for conversion", + ) + parser.add_argument( + "--split", "-s", default="test", type=str, help="Dataset split to use" + ) + parser.add_argument( + "--dataset-root", + "-r", + default="data", + type=str, + help="Root directory for input data", + ) + parser.add_argument( + "--output", "-o", required=True, type=str, help="Root directory for output data" + ) + # Special dataset types + parser_special = parser.add_mutually_exclusive_group() + parser_special.add_argument( + "--retrieval", + action="store_true", + help="Flag to signal retrieval dataset (text captions instead of classes)", + ) + parser_special.add_argument( + "--multilabel", + action="store_true", + help="Flag to signal multilabel classification dataset", + ) + # Additional parameters + parser.add_argument( + "--image-format", + default="webp", + type=str, + help="Image extension for saving: (lossless) webp, png, or jpg (Default: webp)", + ) + parser.add_argument( + "--max-count", + default=10_000, + type=int, + help="Maximum number of images per TAR shard (Default: 10_000)", + ) + parser.add_argument( + "--max-size", + default=1_000_000_000, + type=int, + help="Maximum size in bytes per TAR shard (Default: 1_000_000_000)", + ) + args = parser.parse_args() + return args + + +def main(): + args = get_parser_args() + run(args) + + +def run(args): + # Setup dataset folder + os.makedirs(os.path.join(args.output, args.split), exist_ok=True) + # Load original dataset + dataset = build_dataset( + dataset_name=args.dataset, + root=args.dataset_root, + split=args.split, + transform=PIL_to_bytes(args.image_format), + download=True, + ) + # Run conversion + if args.retrieval: + convert_retrieval_dataset( + dataset, + args.split, + args.output, + transform=None, + image_format=args.image_format, + max_count=args.max_count, + max_size=args.max_size, + ) + else: + convert_dataset( + dataset, + args.split, + args.output, + transform=None, + image_format=args.image_format, + max_count=args.max_count, + max_size=args.max_size, + multilabel=args.multilabel, + ) + + +def PIL_to_bytes(image_format): + OPTIONS = { + "webp": dict(format="webp", lossless=True), + "png": dict(format="png"), + "jpg": dict(format="jpeg"), + } + + def transform(image): + bytestream = io.BytesIO() + image.save(bytestream, **OPTIONS[image_format]) + return bytestream.getvalue() + + return transform + + +def path_to_bytes(filepath): + with open(filepath, "rb") as fp: + return fp.read() + + +def convert_dataset( + dataset, + split, + output_folder, + *, + transform=None, + image_format="webp", + max_count=10_000, + max_size=1_000_000_000, + multilabel=False, + verbose=True, +): + """ + Convert an iterable `dataset` of (image, label) pairs to webdataset (.tar) format, and store in `output_folder/split`. + + Images may be passed in as either: + * File paths: pass in `transform=path_to_bytes`; + * PIL images: pass in `transform=PIL_to_bytes(image_format)` where `image_format` is e.g. "webp"; or + * Raw binary data: use a PyTorch `Dataset` that supports `transform=PIL_to_bytes(image_format)`, and pass in `transform=None` here. + Be sure that the transform is not applied twice. + + Copying image files directly or writing raw binary data is fastest since it allows multiprocessing; + passing in PIL images will be slower, but should work for any format of dataset. + + Labels must be zero-indexed integers (for multilabel datasets, labels must be arrays/tensors). + + Classnames and zero-shot classification templates can be provided as attributes of the dataset (`.classes` and `.templates`) + or filled in manually afterward. `dataset.classes` should be a list of strings indexed by the labels, + and `dataset.templates` should be a list of strings containing `{c}` to specify where classnames are to be inserted. + """ + # Create output directory + os.makedirs(os.path.join(output_folder, split), exist_ok=True) + # Multiprocessed dataloader, should work with Dataset or list + dataloader = torch.utils.data.DataLoader( + dataset, + batch_size=1, + num_workers=8, + collate_fn=lambda batch: batch[0], # No collate, only for multiprocessing + ) + if verbose: + try: + print(f"Dataset size: {len(dataset)}") + except TypeError: + print("IterableDataset has no len()") + # Save classnames + if hasattr(dataset, "classes") and dataset.classes: + classnames_fname = os.path.join(output_folder, "classnames.txt") + with open(classnames_fname, "w") as classnames_file: + print(*dataset.classes, sep="\n", end="\n", file=classnames_file) + if verbose: + print("Saved class names to '%s'" % classnames_fname) + elif verbose: + print("WARNING: No class names found") + # Save zeroshot templates + if hasattr(dataset, "templates") and dataset.templates: + templates_fname = os.path.join( + output_folder, "zeroshot_classification_templates.txt" + ) + with open(templates_fname, "w") as templates_file: + print(*dataset.templates, sep="\n", end="\n", file=templates_file) + if verbose: + print("Saved class names to '%s'" % templates_fname) + elif verbose: + print("WARNING: No zeroshot classification templates found") + # Save dataset type + if multilabel: + type_fname = os.path.join(output_folder, "dataset_type.txt") + with open(type_fname, "w") as type_file: + print("multilabel", end="\n", file=type_file) + if verbose: + print("Saved dataset type to '%s'" % type_fname) + # Write to TAR files + data_fname = os.path.join(output_folder, split, r"%d.tar") + sink = webdataset.ShardWriter(data_fname, maxcount=max_count, maxsize=max_size) + nsamples = 0 + label_type = "npy" if multilabel else "cls" + for index, (input, output) in enumerate(tqdm(dataloader, desc="Converting")): + nsamples += 1 + if isinstance(input, str) and transform is path_to_bytes: + # If copying file, determine image format from extension + extension = ( + os.path.splitext(input)[1] + .replace(".", "") + .lower() + .replace("jpeg", "jpg") + or image_format + ) + else: + extension = image_format + # Convert label if necessary + if isinstance(output, torch.Tensor): + if multilabel: + output = output.detach().cpu().numpy() + else: + output = output.item() + # Write example + sink.write( + { + "__key__": "s%07d" % index, + extension: transform(input) if transform else input, + label_type: output, + } + ) + num_shards = sink.shard + sink.close() + if verbose: + print( + "Saved dataset to '%s'" + % data_fname.replace(r"%d", "{0..%d}" % (num_shards - 1)) + ) + # Save number of shards + nshards_fname = os.path.join(output_folder, split, "nshards.txt") + with open(nshards_fname, "w") as nshards_file: + print(num_shards, end="\n", file=nshards_file) + if verbose: + print("Saved number of shards = %d to '%s'" % (num_shards, nshards_fname)) + print("Final dataset size:", nsamples) + + +def convert_retrieval_dataset( + dataset, + split, + output_folder, + *, + transform=None, + image_format="webp", + max_count=10_000, + max_size=1_000_000_000, + verbose=True, +): + """ + Convert an iterable `dataset` of (image, [caption1, caption2, ...]) pairs to webdataset (.tar) format, and store in `output_folder/split`. + + Labels must be lists of strings, with no newlines. + + Read the documentation of `convert_dataset` for more information. + """ + # Create output directory + os.makedirs(os.path.join(output_folder, split), exist_ok=True) + # Multiprocessed dataloader, should work with Dataset or list + dataloader = torch.utils.data.DataLoader( + dataset, + batch_size=1, + num_workers=8, + collate_fn=lambda batch: batch[0], # No collate, only for multiprocessing + ) + if verbose: + try: + print(f"Dataset size: {len(dataset)}") + except TypeError: + print("IterableDataset has no len()") + # No classnames + # No zeroshot templates + # Save dataset type + type_fname = os.path.join(output_folder, "dataset_type.txt") + with open(type_fname, "w") as type_file: + print("retrieval", end="\n", file=type_file) + if verbose: + print("Saved dataset type to '%s'" % type_fname) + # Write to TAR files + data_fname = os.path.join(output_folder, split, r"%d.tar") + sink = webdataset.ShardWriter(data_fname, maxcount=max_count, maxsize=max_size) + nsamples = 0 + for index, (input, output) in enumerate(tqdm(dataloader, desc="Converting")): + nsamples += 1 + if isinstance(input, str) and transform is path_to_bytes: + # If copying file, determine image format from extension + extension = ( + os.path.splitext(input)[1] + .replace(".", "") + .lower() + .replace("jpeg", "jpg") + or image_format + ) + else: + extension = image_format + sink.write( + { + "__key__": "s%07d" % index, + extension: transform(input) if transform else input, + "txt": "\n".join(caption.replace("\n", r"\n") for caption in output), + } + ) + num_shards = sink.shard + sink.close() + if verbose: + print( + "Saved dataset to '%s'" + % data_fname.replace(r"%d", "{0..%d}" % (num_shards - 1)) + ) + # Save number of shards + nshards_fname = os.path.join(output_folder, split, "nshards.txt") + with open(nshards_fname, "w") as nshards_file: + print(num_shards, end="\n", file=nshards_file) + if verbose: + print("Saved number of shards = %d to '%s'" % (num_shards, nshards_fname)) + print("Final dataset size:", nsamples) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/perception_models/apps/pe/docs/assets/cat.png b/perception_models/apps/pe/docs/assets/cat.png new file mode 100644 index 0000000000000000000000000000000000000000..aaca6ec7d59ae38fcc25e64f10d3876a3dbd2198 Binary files /dev/null and b/perception_models/apps/pe/docs/assets/cat.png differ diff --git a/perception_models/apps/pe/docs/assets/dog.mp4 b/perception_models/apps/pe/docs/assets/dog.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..33f4d9e2bf9c65faceb12304e3d71ef56f0c0ead --- /dev/null +++ b/perception_models/apps/pe/docs/assets/dog.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:fbe6a3d5d0f718ceb453faf53c5bd3a5e3e27db6bbbf5fcef4f61149936307f8 +size 182022 diff --git a/perception_models/apps/pe/docs/assets/dog.png b/perception_models/apps/pe/docs/assets/dog.png new file mode 100644 index 0000000000000000000000000000000000000000..04a8373a7a0a7cf20e1d845e761fdd5f0f8a15fb --- /dev/null +++ b/perception_models/apps/pe/docs/assets/dog.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:0878d74670d0466404341083109b83b08ec61a14a2c00618fb4e235d1924aba6 +size 982442 diff --git a/perception_models/apps/pe/docs/assets/office.mp4 b/perception_models/apps/pe/docs/assets/office.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..964a8d674729b24cca6d54b7bff5e02706d3aa61 --- /dev/null +++ b/perception_models/apps/pe/docs/assets/office.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ea2569275db0ec19de75a0cd081b259646d234b98e216a95e4780b858eb11aee +size 7176599 diff --git a/perception_models/apps/pe/docs/assets/office.wav b/perception_models/apps/pe/docs/assets/office.wav new file mode 100644 index 0000000000000000000000000000000000000000..245138bab5aa0bb5dc0be983ad5b3a03dc9443c3 --- /dev/null +++ b/perception_models/apps/pe/docs/assets/office.wav @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4614dd3b5a31322a9e95afd56aeca33160b6943dda56be620c821c68d0e184fe +size 5759838 diff --git a/perception_models/apps/pe/docs/assets/pikachu.webp b/perception_models/apps/pe/docs/assets/pikachu.webp new file mode 100644 index 0000000000000000000000000000000000000000..7ee88b2df256e830bfc7415e9798137615a461bc --- /dev/null +++ b/perception_models/apps/pe/docs/assets/pikachu.webp @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f5e7f0e624d6a20f67a6a97ee576d6ef44cdb1c2ecefcd1716f6b90a5489c419 +size 2062082 diff --git a/perception_models/apps/pe/docs/assets/shark.png b/perception_models/apps/pe/docs/assets/shark.png new file mode 100644 index 0000000000000000000000000000000000000000..e862a5248e72e00c76938868807e06794cc7fbda --- /dev/null +++ b/perception_models/apps/pe/docs/assets/shark.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3ac46356dcb24035ef9c475b1a0f7824cbefcdc3d6794632c05c08daa850d9ec +size 530770 diff --git a/perception_models/apps/pe/docs/assets/spatial_correspondence.png b/perception_models/apps/pe/docs/assets/spatial_correspondence.png new file mode 100644 index 0000000000000000000000000000000000000000..b3f9f734f966bd64b9d9ad345e6cf657a84443e1 --- /dev/null +++ b/perception_models/apps/pe/docs/assets/spatial_correspondence.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3fa389185f5edf6f581274ced8ecbc4d4b0753428b5f555a4cef7c8c29f79848 +size 2131734 diff --git a/perception_models/apps/pe/docs/assets/spatial_features.png b/perception_models/apps/pe/docs/assets/spatial_features.png new file mode 100644 index 0000000000000000000000000000000000000000..413e5711015e2d7cb4a6a31f654400259b7146fe --- /dev/null +++ b/perception_models/apps/pe/docs/assets/spatial_features.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:2cd8ed39a3091ec68121ae0dc82408cd811050771128a4812e196a02d9a7ed12 +size 1992530 diff --git a/perception_models/apps/pe/docs/assets/teaser.png b/perception_models/apps/pe/docs/assets/teaser.png new file mode 100644 index 0000000000000000000000000000000000000000..d39981d5f87bb0f3b93a59a9c573c1191964b5a9 --- /dev/null +++ b/perception_models/apps/pe/docs/assets/teaser.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:685fd7f4bdad99ecc679d6d5cfb2616e04ce6e66d0d69ae8f95b0a077c890f7a +size 247818 diff --git a/perception_models/apps/pe/docs/assets/train.mp4 b/perception_models/apps/pe/docs/assets/train.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..19bab27036c236586bd937dde57338f278993e1c --- /dev/null +++ b/perception_models/apps/pe/docs/assets/train.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7427d1b493b9c3dd4d1d59399fd0f2ff47a62875ff23aee401e6d53dc7cf94c8 +size 4092764 diff --git a/perception_models/apps/pe/docs/assets/train.wav b/perception_models/apps/pe/docs/assets/train.wav new file mode 100644 index 0000000000000000000000000000000000000000..c16282a8e3678d3f1acb6fe4b10877c1e64def5f --- /dev/null +++ b/perception_models/apps/pe/docs/assets/train.wav @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:90fb58a4a51b91676ee00db19ec6a744688764243e1a153d0b59a805394007e9 +size 2699342 diff --git a/perception_models/apps/pe/docs/evaluation.md b/perception_models/apps/pe/docs/evaluation.md new file mode 100644 index 0000000000000000000000000000000000000000..58201a904d1e1d578c2c790c8970c695dc41d176 --- /dev/null +++ b/perception_models/apps/pe/docs/evaluation.md @@ -0,0 +1,46 @@ +# Zero-Shot ClipBench Evaluation +Please download the supported datasets directly from the datasets host and update paths in clip_benchmark/datasets/builder.py. And run +```bash +model='PE-Core-G14-448' +DATASETS=./clip_benchmark/tasks/wds_benchmarks.txt +DATA_ROOT=DATA_ROOT/ + +python -m clip_benchmark.cli eval \ + --model $model \ + --pretrained $CHECKPOINT \ + --dataset "$DATASETS" \ + --dataset_root $DATA_ROOT \ + --output "./benchmark_{pretrained}_{dataset}_{num_frames}_{model}_{language}_{task}.json" \ + --force-preprocess-cfg resize_mode=squash + +``` +This script will perform zero-shot classification abd retireval benchmarks defined in clip_benchmark/tasks/wds_benchmarks.txt. Examples above includes the following tasks: +- ImageNet 1K classification +- ImageNet v2 classification +- ImageNet Adversial classification +- MS-COCO retrieval +- Flickr30K retrieval +- Kinetics 400 video classification +- MSR-VTT video retrieval + + + +# Zero-Shot Retrieval for PE-AudioVisual + +```bash +python -m clip_benchmark.cli eval \ + --model pe-av-large \ + --reweight-scale 10 \ + --dataset audiocaps-audio-video audiocaps-audio-text audiocaps-video-text clotho-v2 \ + --dataset_root $DATASETS \ + --output "./benchmark_{pretrained}_{dataset}_{num_frames}_{model}_{language}_{task}.json" \ + --batch_size 4 --no_amp +``` + +This will run zero-shot retrieval for the following tasks: +- Audiocaps Audio-Video +- Audiocaps Audio-Text +- Audiocaps Video-Text +- Clotho-V2 Audio-Text + +Clotho-V2 will be downloaded from its original source and unpacked, but due to Audiocaps being a Youtube dataset, the user will need to provide the audio and video paths under `$DATASETS/audiocaps/audio` and `$DATASETS/audiocaps/video` respectively. diff --git a/perception_models/apps/pe/docs/pe_demo.ipynb b/perception_models/apps/pe/docs/pe_demo.ipynb new file mode 100644 index 0000000000000000000000000000000000000000..4cc359be27cd9b36194402029c0430d5009f43ea --- /dev/null +++ b/perception_models/apps/pe/docs/pe_demo.ipynb @@ -0,0 +1,289 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "e6f42b3a", + "metadata": {}, + "source": [ + "# Perception Encoder Demo\n", + "[![Paper](https://img.shields.io/badge/Paper-Perception%20Encoder-b31b1b.svg)](https://ai.meta.com/research/publications/perception-encoder-the-best-visual-embeddings-are-not-at-the-output-of-the-network) \n", + "[![Paper](https://img.shields.io/badge/arXiv-2504.13181-brightgreen.svg?style=flat-square)](https://arxiv.org/abs/2504.13181)\n", + "[![Hugging Face Collection](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Collection-blue)](https://huggingface.co/collections/facebook/perception-encoder-67f977c9a65ca5895a7f6ba1)\n", + "[![Model License](https://img.shields.io/badge/Model_License-Apache_2.0-olive)](https://opensource.org/licenses/Apache-2.0)\n", + "\n", + "This notebook provides examples of image and video feature extraction with pre-trained Perception Encoder (PE). These featuire can be used for image and video zero-shot classification and retrieval.\n", + "\n", + "You can run the demo locally or run it on Google colab. You can also run it with (faster) or wihtout GPU." + ] + }, + { + "cell_type": "markdown", + "id": "75e90ff6", + "metadata": {}, + "source": [ + "### Prepare Environment" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "89375a2a", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "GPU is available. Use GPU for this demo\n" + ] + } + ], + "source": [ + "import os, sys\n", + "import torch\n", + "import matplotlib.pyplot as plt\n", + "from PIL import Image\n", + "\n", + "# check whether run in Colab\n", + "if 'google.colab' in sys.modules:\n", + " print('Running in Colab.')\n", + " !git clone https://github.com/facebookresearch/perception_models.git\n", + " !pip install decord\n", + " !pip install ftfy\n", + " sys.path.append('./perception_models')\n", + " os.chdir('./perception_models')\n", + "else:\n", + " sys.path.append('../../../')\n", + "import decord\n", + "\n", + "if torch.cuda.is_available():\n", + " print('GPU is available. Use GPU for this demo')\n", + "else:\n", + " print('Use CPU for this demo')\n", + "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")" + ] + }, + { + "cell_type": "markdown", + "id": "6e04507d", + "metadata": {}, + "source": [ + "### Create Model and Preprocess Transform\n", + "\n", + "The following code creates a PE-core model and loads pretrained checkpoints. Then it gets the preprocess image transfrom and text tokenizer. The models available are:\n", + "- PE-core-G14-448 \n", + "- PE-core-L14-336\n", + "- PE-core-B16-224" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "94a3c87c", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Missing keys for loading model: []\n", + "Unexpected keys for loading model: []\n" + ] + } + ], + "source": [ + "import core.vision_encoder.pe as pe\n", + "import core.vision_encoder.transforms as transforms\n", + "\n", + "model_name = 'PE-Core-G14-448' \n", + "# models available: ['PE-Core-G14-448', 'PE-Core-L14-336', 'PE-Core-B16-224']\n", + "\n", + "model = pe.CLIP.from_config(model_name, pretrained=True) # Downloads from HF\n", + "model = model.to(device)\n", + "\n", + "preprocess = transforms.get_image_transform(model.image_size)\n", + "tokenizer = transforms.get_text_tokenizer(model.context_length)" + ] + }, + { + "cell_type": "markdown", + "id": "8587d072", + "metadata": {}, + "source": [ + "### Example 1: Image and Text Feature Extraction for Zero-shot Image Classification/Retrieval\n", + "\n", + "In this example, we extract the embedding of a cat image, and the embeddings of 3 senetence. We measure the cosine similarites between the image and sentences." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "38f80aa1", + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAASkAAAGFCAYAAAChRwUXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8ekN5oAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOz92ZIkSZYliJ3LzCKiqra5e0RGLrV0Z1U3iABQDxFe8YAH/CA+At8CIhDwgEfMEKEA9Aymc6pyiXB3M1UV4eXi4d7LzCKq5m7m4ZEZRRScGW5mqrKwsDAfPncnZmb80n5pv7Rf2s+0ub91B35pv7Rf2i/tU+0XkPql/dJ+aT/r9gtI/dJ+ab+0n3X7BaR+ab+0X9rPuv0CUr+0X9ov7WfdfgGpX9ov7Zf2s26/gNQv7Zf2S/tZt19A6pf2S/ul/axbeOmB/5f/8//p2e8IADMD3P0NgLmg2AeQDwkArviPFv1s61u6/rt84jv7rNTPmVmvW9bn8PXz5e/X+baursHdWHR/A7zp+fqYy72CAFA7hggMBxAB1H4SDQjDDsPuFm5/Cz/egIYRTEAIAcMwwAeCIwdygHeAIwKBgZIBZpSUEJeIj4/v8d//D/8P/N//b/9X/C//8x8xIOCbhwN+9XCDb+5v8e7NPcJuhB9GkHNybe9BRNpjgnPtOYjqw7djiLBuDkQkQ7D5iohgX7D+ByIZK9L7ODmXSI4n7QfYjtexo2tj7PVEgEnOdNT3X+7F1K4jY+6eeRbos5B1SPrm9DNcGwPtsd1X+8LddyA7h0AgMLE+Cmt/pJ8Et77Wdiwv7tn3p90LV55rO27bZ782Fp9rTs/5P/wf//efPfbFIPWj2i8+7S9qAkodoj/TSBcCXfnPOQfyAiLDMAhYwObe5pos/zDLNfe7Pfb7HRiMp/MZYwD2U8DNfodzShgXhwDCMIwQnJMNwTkH5xyY+Ysm7Fdp/dD9nJoN+Sf6xswvAIcvv/6Pbn+Ne3yi/XVACj/P+fNzao2RrWdE3emI4ByhwIGcA1EAkfwOdwlUxnLq6TB2WeQeLP9x4XrctNvh9vYeIYx4jB/x4bRgelqw30VM+wg4jwIHUILzDt55FBaOKIyBBbTIfUXM+LRGonEU/GyByiSL9Sd9225OL32IjgW9pj9fBIjubza+fxOQ+usQK+7++yvd7uUfv+6yOg+ZhPJ78oBzIBfqf845UCA43zZlIgKoYzfM4NWYCGARAT4EjMOE/W6PcZrA9ITjkvD+OGO3XzDtI8h57Ce9JoBhHFYiXu0z8yeJgfXA0fPjw1Ax7JosqK0HJ+rX+Bc1Uwu8ftH3V9je/hKg+m9+TBOx9rUMrBfznuvvs/f7G7WvAlL9lN8+yk/xaD3rYN7qqdox/FOClIpKZCou5tWElPt/6fObroXqXDRgaIo/ByIPIge2Y1QJJvOwoJQC59q1ZGwKmIvqpYwFAcF7+BCw2x0w7SbAAeeY8HGesT/OOOwXBOdAzABGOZ8Y4zgCcKoLlHuz6U/Y7tveARteSk/QjhZAqrqkqtB6boTQ7ey2y1P3bf/7tWZzQ98a2cC97q1V3rvVqW3+Iqb2Dl50+U3/rjFK0mPIdcf06+EScJtoqde+1hfefvi3padfjUnZhld1vVj//Ckac73zlf78FRjUxlBQFfLPwOOnxmJLwRmq8CVTnrZnYlPyktEm1x0vc5YVwJm9AIYBqoI3qdhHKvYROYzDDre3d7i7vYUfCJwKzsuMp/MZj8cZkyN42xRoADlCCB40hKrzqkDVAZQ92yW3lW2NVrOlVxxfH7/+J/ffVqB6KUh9CpReaUC57JACKVUd4hpIP9MuZNiOLtbrbAFnq9t6HhGvGitWv/185OavBlLP09q/dvt59AJAXTdKcKQ9JxauLI064VYTRSa6MRDHpm8ScKqgBO42wvXNKkCw6I5QClAtoMA07nF/94C7+zsc9jsc5zNiSjidZzwdz9gTwes5zgHDEJBzQSny9xagPkdKXrNmG2W5/Lx+9Jp19cljC77EO+fTC//rN0I3xl94o540/aSEovv9tff5KiD1EoD6+YDY12mVHbw6HddLRqKtbpF8zMRsE5MFGIztM8ClIC0JcBHOK8u6dmVWFOGiYJVlt2fCEALubm7x9uEeNzd7/PDhI84lYp4XASpH8ChwjjFOHlx2gIp1JavUATVvq76EvvCtryZyr5j7mvv9T0wWXgXCr2xrlvbT3GPdTKT8Ea0jgq9pLwapfjFe2cy6X9j+35/cdDVXrre9x6e+syv0Pk3Xjr8GBV8rvx8RgZwDJxGZar+3d+Trv7cxWP/S/m3Lz5gPCHDkUFRkQ0ngElRHJECVS0aMCWEYxJ1IrXZV/7zpHEP1aMqAhhCw3+/x9s0D3r15gz9//x7zHLGkiOP5LOIeGM4zDocJuZRKz6RPADlaLUzGmhSuCePVjq0HqYq58kYN9mh9iJ7SXY/aUZd6oLZaeoV7PZXaPdZiIVfxtfmw9T3oum/HdTOfuv78GFcN6ruFNqb9lnDtcT/ZdCOoc3D1Xky0lCOa+956o75UWTx/r9eoY14OUtbdT1y7Dt4VMWP1N1928csA6tMPKtIQVXHma1E5Uie6Qs0J9Rooig4ITedSF4SCEJe6mHQJVMWmuSaybseOIBY905IzI3EWHVEpClYELkBJDPZcnf6YbbctABdxG2DI/UsBsYiQwTkcdju8uX/At9+8w//03/6Ax+CRY8YpLvBnAiHD+YKbZY/bJaFMCRwCABI/U2YQt8VpO33nQ1sHgszTl9xqITldgHXBWN9Nt/YCftb0lXShB5Y/XSf+9i+nacg6GEcFSTKgepk4KMdyfb/6hC86t/W2bVp25poEtP7avHkOPD/d1/6MKzssCxM3g86Paa9Ziq8T9z5z5Z9CpPsS9rOyJgGrne851H/pdYVFUb3mxSatrZ/a60+tV23HXn1PvSLZjjFLXhC/KDiUosa5XICcQDzCVmPOGaWo532dT4wCRuHc9FGFASaxkKoiPYSAw+GAd2/e4vb2Bu8/PiGniDlFeW5keA/cnRfM84JlGjEMI5wrYHbmfgUmrs9hQHs5IHTx+8VYPjfA3cHrQ17wXp9VYrX+fl0xTXv4xdd8wdjU32l1zpfe66u2nuB+welfTScFfH2A+tJW+2Eg8sX6oyvXtk2lqHXs1X1bzypjGgQgg5SpdqEfZOEaHgUeRB7eD8pWPJg8TGlOYDgFz5wzfPBw7EDORFL5rhQDqQwqBHAGiRwJYmA3Tnh4uMe3b9/hj//2F5wQETODUwZxwRAcjsuCJRcskTEl0VMRATlXlELxanektaj20vZTbHo/ZXuuvz/FM/xV1FA/YfvpmNRXuOFfs1Uh8avWmhCYycxfxvIAk0MbXHXqDdGr6HeksVg+gBFQ4ODIw/kA70eAxPWAPClzKXYH5JIlbMWxfFc90h3AWYmTugiwskQUkAN2+xEPd7f47lfv8F//xxHvH4/IADgVnJgwLAmP5xnHJWEXI8Ylqq8WIQSHooBEDBT173KfWVbbkJprOqe/ZltJgC9snwTVr/wQf8ux+ZL2V2VSW01Qk5ZfcG6vQL8a3PvMd8V8c1Syr4qE1pvyCcDog4q/RivM4FLqPZ1e/qLfAAoai6jsqbqEd6IQsAIn+VqDdv0AxoBCBLgAOA8KA1zwICciIsOJzopMb1KETXlfxVMDPYIDw6x/uVuQBQUF5Dxubm7xcHeP29sD6M/vkTMjFwYKI4aC81LwdJqxG0aEYQAFB8oRLohl0SmImw7TNG9VBKQGTGWjjO7nE9eVTz9OBLONgC5/t+evb28jhX7yvr0k96xo+ppV8ulbraU6/aRe2qM5c9qHf71EJ59aYs9K+S9orwOpKxe39f+a+36KgXwyC0LV7H3imKsXlR9falG51qetWvHqlXvlrS1QUrZUVQftbGJorIj+TQCRg/cD2O+QATgvTIrIw4UBPohDZ1ER0XmPfk9nSHyeqYA7Ktd1U36vOisUOO8x7Sbs9zt4J30rRXRNqRSc54jHpzP244BxHDGMAeQJLicMqggXEGD1R216lW5YmmKZO7H8ynHrIX1dIPPqyCv6pi041X1j+/enrr/qdH9Th68FUtbqLai+VW0FsM3nb0GxPjdG2rZE53Ptrxa79zdttmP2bhSfmeTPAZ+xMjam8CWzoddtrkxPDZwkjYmlBiF45+GUFRUiSelBHuQnCThWGJJYOlkURKbIZuRcNvriDcySgFnh5gy6GyY83N9hGkaczwmsOqWcgTkmPJ1mHHYTxumMYTeAPIGcxPgF70HMAnBGU762yMOm1xMw/tFs61r7miJff9TXlte+9Do/V11N174KSPWE8uf0zH1YxhezN1y6GfwYHRerfCGe2Q2tFJdUd7MGqFIKuGQBKSIQfBPbTIyDg3cO3qs7gJ7vyJifuBoQLgG6xttpPCAoI3gvPlP3b/Bwf4en03uURZTipRSkVDAvCU/nGdN5xDSeRadFgHMRDgRPTvReKpLUOMcmr7yqEZE8H1FNE6Nc8dlNh/Hj/ZJe3L/u56Vv1vZIwjY/2qvvZfPoSy9S3V3wSdeir91eyytf7cx57eLbNdt0Rpfnfw4QXtM+dZ+Xfcara7wUrHB52+fvaUBkA1d1NKboNh2NAA05EdtacHEBkEElirI77OCdHgeAi0w05zx8GOB9qMnbBJwLavCvLujnZgg50rxQDmEYMO0mHA43uDkcMPiPiIgAgFwYS8qYl4jzWbzRp8GDiOG9QwgBxYvOzrHplUzclVXFZGBtgchNR3VtLKlbRTln5Jwlmd/WgdDmWafLMrGyjokJHGsFTy9HVZCh1jn9nGtf++tZfORKjGauq5/q3KFXY3Rj7Ar22Gy+n7lgr5zoHUrr83R9fw7q2wU6VQJ3H20O+3x7+QC8gklx7cSnLv8cIPxYBvKJXn36+05pfhGR/xng5M1nvStD4QLabGMXjI2vBBMoKJIpOsiJ0tv+dgF9hkcmgLiASwQywGWAMxGPCKwWOi4FyEUU545rChVZUAWicWLNiKhglAvWS8fBk0Mhh0zAOA64vb3F/e099tN7nE+x6rhiyphTxmmJmE4zggNABcPgsdtNKKWg5CwuEqwLyyjGF/moyc+iPmDXl1MDicZkWD0iTCfWeX7bk3f7SNuzSPzIOoASl/ouu0N9t7S67qpv3J6ZgNW9XsSjuk2t0iYDzE8wyO0l2nvu+tcB1GevUzeJvHl/vfLu5e/1NVPg5SD1c5Lj/grt2uO23atcHPQp66S07VtRUYgEHMxQzxQA59BpkGTpMQM5g2MCKME7eXVERY0XhJSSHO+85JcikbG2IUQEwDmHwk6c0PteOQKzgyOHaZrw5uEB337zDf7b//xvOI5HLDFVP6w5JsxLwvm8wKHAO8bNfkJKCcUHZOcQiqaS6Yeh0pQrLKpjLNshIzRLoDz7CxboSsT8d9Reu94aCr36xKoz/KLzf/r2YpD62/d9+9b+NqjpLK6L1/mjgM8xRRtB1/1teiVJWieuBSPgrgQH686Z1QkTpcCjEbDah1Ig/pSsOiYJvTGxiovolBxxtWCJcr3tid4REDxS8rjZH/D2/h63Nwe8f/8eKQEZ4lpRGFhiwnFZkEvEbgrISZT0uRS4UpByQoCv4ulqMbC7BCgTPTYe6gZIkgmUXgRQP+/26fn7OQlq9fS8/eDHtJ8fWv0okLoYm0+wic8zjc+1XjTjT17vJXqvraj3El1WKzoAcP/vi3VVRrQ1ab5zID/C+SD+Tz7AD5P4O8E13QlRW8yg6nbQ6yJEfy4w0/RhXEGMGUBh5JRQSoYj1tA+GctSVPdFIhL6EOB9wDTtcHd7i9ubCU4Sgko0jSNkIpxyhosJJTFOc8RpWbBbFgwhQDIbh9o359r4tfEirNC+X53dgVW86qTF537/bPvMvb5uu7a5FvQpDJ+7tZFOE8eu6ezqc/OGWfIV6ctupjuS/NkP8moHuX7u9pH+Clj2o617lnmnX6hbx8ovBygTU8yZE90AXddzcdn8fQU1TFzrAepiKm3Ay1KktBd1BeReKJM35TVDBDsGOYgneZgANwBhBDvpq6qu7GSx+mmeYIKEzTjnRPFNWjeEGrg1PU3W2L2CXMTiJwGABaUkgDMyihRSYYbXpHaHww53tzeYxoA5RpTiUAhIOt6n8wIaHM5RQGo/LwjegdwEpwBFjkEF4h0P8UZ3xpJM+kP7u6n7nFo0nfh7Wcpcpgq+VHVP110QthqU/nd6+Wv7wsbdf/oeTLdpKibqLOQXYKvs0vRHrAPHDFgFoSrPUneyfc7dNUkNAe3qsn7NMGEbxjUHUEO1a5WNftr2VVwQatkq2+qeaV+sOKd6F7vSy6+33RBecM51ZrTZxTpgW1/z5ds5oyBzAmebGwOQM5wbEXwADQGsYXyOHCzUpO7CBAAOpP5TzonZ38oF2RZrPl0MEQeLiou5ZFAR581SEggZBRmlkIh8LOLtzWGPd28fcHdzwNP5hNTt7IUIS84YA+EcE47zgt08I3gJkQme4BH0aQke60XUK5K3o9f2ddVfFXvvWIHac1bBvtVvX8O4vlqrFKb9NEfM7QDg8zOoicd2wjUYpvVafO65Sd6MQJ762G0TLlYQtEfYXuinHdAfD1KfE57/vbcr48+mxPnSCwiBVx1RgXMS6OvijMKE4DyoDPDwCGEAOVFkexfq1lt3Y6A6NXt1IXDGoBhVlMuclYmoCMEMFJbcd2C1xiUQEoIjFC+gyBAv9/v7Ozy8ucdfPrzHkpMNBOQyBUui6pYwzwlzSJjGhHEImpHh2iKhJu7VkVmP02p9o2dY11rPWP7qSPSi1nr48v5dJGn9ik1Y608q7z7bXkpavkgn9VPg0fN6pF5UpPrdc+LZ125VD8XtPqyLc/VaP/mON1+SeoMT1ApnBRcLclnA8YziHIIDyDmEQfyifBikUkwFKWUWmoGAnINzVCcd51yHL2cbN1nALXkcwZxHhQiLQpy01JUjh2EYcX9/j7dv7vGvf9zhaXkCSoG4Tsg1U8qYl4zzola/mDAvEbtxRAkFCI39ttTCBiabd68fi7Qifu7t/X5Kaf63WWyvaaY8+FQ/ex+sNu8+ecpnWz9mzaG4saO/Nrt8zXr9osycz17+BQrr5z/vqDzam1mXNOBqUr7Qc3Ug8rn7rfygnhXbNs/EuDhuBaPPiroqcnXWKyIR0Zw6bsKqvijVzmUBZw8sroa5kFr8fBjgyIvXE5emWFHdj3NNyS7inRZd6JT+kiuhKIlhmEgoIpRVDIakgyGHEAYcDjd48+YN9vs9xsezpUev66YUsfQt6uC5Cx5xCFhiwjiKY+fK1wzmUNjG1/Cqrkeq7qeqm3uOtl+JLNgsOhORRa9DL1iVbT7WV7d5pwDqe62KbTQP+170Ekym7dU7UWqjMiCbZ58TY23ANA0Odw7Cm3Fq/lFYfW/hU62iEHXnXLZ+nTRH0G3Hrvd7RXa+NpN67nIrhvHSi127Pm/B6PpE7PBo1blPA+IG0K78/nzHnpkm9r55O4G319RJU1++gJQoui0flACWHEIo0IzmnFFSRHEOcRGHBQ8ChlH8nFzXM2qLoC1USeFiuivnPAolUP2MARLHVOllc49gNl0jwXmPcZxwd3OLw26H3Tgg5oiSC8ASyCPEjsQl4XzGbnDYjwG55Lqp9JsLEUnGBRX5jDXULAm2sL0YBEoxuLrGQkrts+WuWr2RDeWtVlDVCV0ys/6l92E1HfOzeViKvrN2b0JfuqqdZ8cYU0RpcLJ6kb1s+4lGOn41yNgAigCG64IdOnDc6E/NM795/vdz9Vofthu+ARW6+30aWpv1+ZOPV9vPKMB4DSZXj+DtLz9hV57tw+vvbYxNrHFooKo7IKn5jgxMyEFyGYinOWcSvQ8XlBwx5AnjMMKFAVzN+gZQ63lFagGUTC4OPgRw0iyaEIaTs1j8+opIliaWGXDOYxxH8T6/u8Wff/iA05Kx5FL9xWQnJyxLwtkB59Fj2Y+SYE9fbSmSBoZFUETLftBNduqCTJhxs7/BEiNyjn9d3yh7zT/6lmuFW5+S+Pqlv+zGjaWuKWn/2eXcZTQXEF6zty0z7Z/oBe/h6nnoaEh5+Tr6GYHUa9qKNP7kd/iStiZX3Kg4OWQukqu8sJjhyZiTmdAZJUdQKYBLKCnBD4Nm0hRXAXBBAItlj1R8IV8txKY7szg3QPRWzgmLK9m1vnFRPykGMaMQi6XHco87Bz8EHG4OePfuG/zrn77H8bggqjtDzccOcT+JKWOJBSkWpJiRU0HyGeQCoEHI5nDlOtt70YBqATENJmYgx/RVAWoj0Vxpz4mVr2kbPlHnQMfKLOJ727fuCi/pAa9AqAGgbYhYgdeqS03c74BpK+pePtfn+vPM59y7K/0EIPUy0ejzCu1PO2FeXA6X9NIGtYmGn+vjS5w5G7vvjuX1sZ++XvcHrX/pv6qiR2HAsSipsZ4QpBfkIu7dJSeUHFFCBKZ91XkUAhwHCS52Ht43T3UiQs4ZKUbNeS7uBqUksbapSweDq76IoforVnfDoNZCTwgccHO4wa9+9S3+l3/9Iz4+njDPEYkzChGIPBiScj1lxrKIfiouCUuMCo5BLI+eugXyDJ9ghncOMUbknECaJ+urkqmuosx24+P2a9NJ6Ytqeqc1ezUdnX3Xv1PeXL0aN67o0uQby6FPq070yu6e+bT4u/V1rYPcjV3fl0LtyWvfq7S5zldlnH2lk7K+vxBRq6T3kyjOX3D3VwFUKZvvtvfbUs1SlXrX9EzPofO1wTBn061sTODLflx5huvgegWQsHnJhkamRGYJwnVwcINY5iQ8JrT+AKJfgAMXIKaEQg5MDgUOgQH2AAUHyWLSZoskMhVnTah/VCmlglZRp1aLiWMUcLEy7ARXIMCi2RlM5Hv79i3+9U9/AbknyWag3EfATvRHKRdRpMeIGCXWMIQE7z2cs8rL1xHH1UVC2jvfiZ8mFn8KrTYOhxs3B3s70pqieb3SqP5aQ4c79aLTd8lKXW2dlv49mzKdtqHEDl2aBpiD5npPVvGr6jRZFdybjAu0ZlLN6diA6/oI9UPCFsC52lzJurAasa27Al8JtP9ce63G5OuWWX/psc/2ck2PL79mfQH8uhu+4N6dWP7K6z3zOeoUlAR1UImsfms/lBXqjuicV9YAnWgCSEQWdOzATEipoCABIAQ4JMrwpcj5qvsi71GCB8ei3uuAiXcGtmT9qpadBsIlZ7ATAGUKGIYRh/0NHt484LDfST7z4lQ5XpBLQqGAAiBq8PE8RyxzRHABacjwPsN7B2K/Guw+5Kj+og6c5lrR7d+bn9fa595kY1GmMK6MoAeqSm0uVQwVVKpLxTN36idJ3UTshI0IdtG/69frU8R88il7RrX9jtrz23O3NNesQGbnKhijmjQu+vfytfO6xfvvTyf1UkH9x7TPXH/19bX+GLOqTID0TwI683flWSwVXCSjgXzm/QDnA5i8Bc7AQfyjyInLQimMlApACYEIOSVhY2hT3EJmpHKxU9FMAo8FmGUS225YrLoVNRGwFE1m5z3GaYfbwy3ub+/w5+EvKCWDo1jqxBYgrIdLQYwZp/OC8xwRwogpy3PK9da5oHq/uPqdjovrduqOtF5pupBesdPI8wsjEPsp4Vn68QVtKxpd7GrPAQieIX9XDn9OSd0AlAGyeMHu+a5IP+3k7veVePDjrPif6vNz7d8fSHXtazhwGihcJW74DF598oButzeAgoosdRciSWrHGbkI7SbS7Jzk4YKUr3KkifA0hQs5SR0s1kJTNksyOJecimDSiooA3ktASskFIQi4lW6B9O4LK92KfCpWOiJM04SbwwGHwwHTNGFeFjhXqmgg0gAjMxBTwem84HReME0TUs4YSkHOUgLL7tXu39gNkausowLp595HP+5fAFRrFPhxQNXY8fY+1BY6dTyocHNNQHsXMmcab+H+WlX1QRfdvbyXSB/ykemovij59Y9ur123X6ek1VbH84rOXD+nd7AEevptir1LBfxlH67v1N3vHbvf9qNOiI1kyd0i+KzYWuePq5MCUGucepqziRmcwEwCVtmDKYMpI1CAC5L5QKq+aFUYBJjZn3RBmDWPdVZT1Z+JmOgcC1gRJH5Pn4fhDF0q62t6DtHx2zP5EDDuJuz2O4zjcCWzAQNwwqRSQowZ87xgniPO4ww/eDhPcMWpJXKtP2J9P8ENem/u+rNuvT/YdmGvRLJ+ddcdScGwv6y902esYZ181/qjKoiWOdOtdJR9z1cVcSqA9M8gl2CyfvXPsYHoqu9az7X+kMrUV0xUns8yha4efzOn146apfv0+e1itVyvff8FqPhikHK0vvqnSkh9rpnJnPlytwG2E7KfXd2f6BXYL+uLLYDaD6BblHrM5uWZAr+KAXXH7SeM/dkWi/3LGiFMIHXgDCA3iJOiD20iEFdAKLmAOCGmEyguCOMO4zQgDEE81dkD5ODdKO4CTlhSzbmkXaQicXXik6LFQ51Y4kpRL3cGsrKbrUNuYc0L5ZwUVoC4Dw7jiGkaMQQLF+buqeUKpUih0JgL5phwXmb4s0cY5D8qHr5ApV/XLThRRjMkJCdzrOmQsbpX/wb6T0ykotXH1rN+ka6dHNHebdV+r1vd0MDNcbGU2m95l+Jm0i7aesrdO6pgeMF40K3kHijbQ8h76K2CLVZzDTQtGyqtrreZu3UMNmNiF+yeYc29uJ/y/WGbD9tXfP2rT7Yf5YLwau/tn6Q9f98tw2sMarN9Aqq3EeCrOhNsn8vEk7WJenstiYUTRiEMRkQXyckdtGaeOGNKBRjRwZRSEGNCjAkpZxROoExY4hnL2WOYBozTiGHYIXiAySlAWTgM1UnFzJrxoHR+UkV65MQHyXuPBBETU0rgIhWKTeIozEg5wzsHBHEfCBQwDAG73YRxHEUHZoYBRXsTJ0pmxCXjPEcMxzOGwWNZAqY0IfiCXCDPvlHuMjOGcUBKZb0Q+yH+2TfZGC6BtDuiAyo7wxxaG8HqPN7ZrL02/9zFMav52oH1xfrs2NS6112Pt9erHOo5tmW3vc60uD3qq17jF7sgfA1QaovKQKRjvV8J84wdrACKu0GidhzQg9W1YSaIK0Q1DHcMwkQvXx9CwElASHZRpyEvDnADXNjDDyNIY/c8ABci3BCRlgUxnZDyghIzIhjnMzAOI8Zxj3G8xbC7gR934ugZQmVUBlBchAnlYrk0JSkHFZl83g8YhoSUI5gZKSd4qIOpzu/MBcUr0KnVcRhG3N7e4fb2BtM4Ii6LiITUPMUzMxwVxJwlvbADdrsR6ZCRcgLziFI3BIKnZr0jIozjiNP8uBn7F73xVxzbzli1LwTCS/6/kQI+cV7jXAwLpLd32WctrayfAdFhrn2trob4oL3Q6rpgej5uR1kO93omi7mmv5ZVpK7PxZvxZrR7rM7kVVdes7z/horzpjS1/cEQn/kSBNes7dW3aqqI7VcdNRaGITqjlFqFYtvBeTP1AFQzLZmHNlSEI6gzTadD0r8lXq9Z2pwfQE6YjQcQckQazwjJY5lPiHEBpwjmgrQskmEzM5YYEXYHjLs9hnGHMAxgeIBV0a1AZc+Zi3mIi/juvMMwjsglIaWIGBkpZSjxAxHDO4+cnfRNx2aaJtzcHHB7e4tpGnE6+nrd+n5IAnljLqAYQWfGeVmwJPGdGsYkHuiqnzMxjkAI3uN0PrfQCYJYPvV77hUbJnYpc5E/XpaYbSWM0VYQWi/mrRtBDwhdIE895zna96nrXOtfYy7XAnntfKyAav3YPZDw6p69yqZKmRud7lrHCzR26DpW0fWMqOkbVv0wVxv76OU7wd/Yutd5Pj8DTvbd5pNX3cV0Cc9fV4YsJ0bJWRTUzmnWynZHexUEZekwJugAeDgaAC2kQOoqwGTinsTkOfJACKKPch7kApwbBASGCSF4gDOAHUq+QYoL4nJGirN4X2v6X5DE8+VsHuTiOY4sCvHei7wwryYks3i6M2eAGSEMGIYdlnlBzAtSTnAETSHsm4uCthAC9oc97h/usN/v8OHDR6lkA1SwNsggBlJWRjXPmOOCXRT25RVwHImuTHLXEPwwYl7mTgtCF++qPQxV3TtVinwhgKBW9t18eu3venbnvXlhhDERisyzfA1qYFyWCtoA1Kcab34aT18f1DMow4z1WBkwkQKVlUBrPmGtPyulPlCv23e/cbxSGeDKl2rDrBhbR1b9/BWF/r6aTuo17fK80iZGhfSveI+OETWp73JXAoRtoACkzMFi2lY8uz9F938BogDyIygEkAtgZQqk1KQW33QOzgeM4w5hnESRToTgnGTkdFq2Sul8yQlxOWNZFsRlxhIXwU9ycGFAGActr64GDTbQL5I9gBn9nJTJVYDMcOpdPk0TmG9wLGKRK5zhSgYwqpe4W03icRxxf3eHb759hx9++ICcZmFv0BTGVYnPyExImXGaZxxPZ0zThDAkOBcwOo9UJFBZMouqgaHqUcxwwd3CvlysjVHp2zIx/srh29fOMEbcDdALW2M56/uCPn1b63YDu64z3QK2mLomSuvlN0DSA1X/EMzG8dp1euu2XUvGqY13/XzLrDposge1MSDuwni6Q9rz9Z//FZjUj9ZJMXcoS01E+lrKKBimdNYgU1B+4h7rbAKEEIJmCdicw5AgYXttTlkSeQEqP0HiVEx2AlwY4C0kxBHIDxJ24r1kPvBSntzYl3MS5xYIGHY3GGNEjAvmZUbW8lXQ3FSW8kT6bgVB1/yPC3VsVYKJs3wAABjHEbzbgxWocikouVQHzProzAjeY9pNuDnc4LA/YDknpJJhUq6HLVgGuICLFBU9n2ecz7NWInZgLXTq82YjYQert+WMtX4ObX5EW2WA+NJrAM+HA24bc3cwYLqcyxATA0Fe+bUxc90Iqs8UNmEwxvh6ka9X/HbXauC+vsfl3wZwnUjN/QMziunUijImgrLSBo6ErUj+fPurinu9y0HzcbF/WDt/Tfb+gntdu3d/3/WX6P8HAFxUh+MFqFJKG3bW/8oAU33Zon8KKvqpf5MPCNNUg4Ct+i6zlAwvpKWaHAkLAwAa4BAU4EZMwx6hJPhlRowLiE2s0+FjBSgYe+qCh5WNGKsCRLlOFr9HLQtn8ANiisgpY6GIcRjr7m6iF2ldvt004fZwwPHpVMUis4xSPUMAKicJPD6fZwQFbO8DBu+RcxbWSIScJK6QucDBX1CjCrtXRabXg1XPz37UzOOmnaq4oFf+LKuql9gAQ//LBYHc6LM2xzD6Nec/KWE2cvWZEaiktZMuaP09dD18ajRr7coXtC8W917CpJ71g6r6EqD7Z3XN14iX2/OMQdXxMoDqzrEd25hoZ2TUHwIaEnoSa/nyClYW3kKyHIv5ITGLrqdkrTUnOh1TPHsf4LyD05IEXAN/cwUV7xl+HMBZ8o178nI75wFihGkneaGYJUd5EdaUk7gSELMu8jaDaEOx2ZggdxSeABc8hmlCzAvmlJBU0Q0adNwM8IBp3GG/3+Pm9haHxyOQueFJ2U74oq4OBTEVzHNU62JGDAU+MKgwnFcRQ/NQMdlCvGRSbaOjyiyqiGjvmdYLyeZGlY1sPdV3jna/dnl0F9RDbbWuvmzsiNdfbRd/cymwRHV0cezWI3yrxqHNrQFc5Gkindy2ca3GsLeo6kD0mSmubgEVTe3doLI0RoZJRdaxJuU9f83PtZ+WSX0C2LpHAOoAfoVbXvz8FJp3xxqbWvVDfs8ZKJmxP9yAQcjzvBlt84TPYGSUHCGvpYA8AJZQFfYqQg4DAgVIymDWxHOS6TKlKIpuxwh+0FQaDGInhRYU8MgL2yrqhAlmlJCxLIuIggzkEisLsWh12i5kBkAsDqRO4vOGccSUD8ipqHi5CLA638pQwWH0A3bTDvvdDtM0Ic5LFY0l3EeO7LcHZmBZEpyLGEJCHDLCkOBTgCOPcVCRVHVpZLHWIMOUer0+FQmoq27Mqma+pvvo9DJrb+2eGm+V84YhhJ4VGSM1gBFP8U4S6J7/GpfqV0NVM9T+2HO3dnEFvpDcrjej2r0oSeufsllvGNBaySXjVQnQduPvL7yOy7QntLF77Vr/m1j3riutgU/Rw6/eh4u/rwOlbT6FC2KMmMYRXApSzsharAAOEnvlGFySpIoyx8iSpEZe8UBewGMSW+C4l/AQ1T95F1C8+CzlUpDiAs4ZJXu45OG8B/OAYRq1Gowmr/O6ezKjuFR1aQwGslgri6a4JYjeTJzqcn3GosBCADwRhmEUkFBHUEntIgnqfK1EzAhBiofudjvsxhFzCOAi42I7erW4KWtlzYUOzBiHEcOYEKKkcQmu+ZjlLC4NBUX8s5gb6Dy7MDWTA9kCWi8yABc5yfU0rPQqpJubflZJ12r9XukEy3nGR2BjQB1w2d+mL7oC5JWhMXdL/1IN0q7Dzy8dw0vJNrfRWW2P/byqpWeoFdi2L4SurPEODF9LSn4mAcZbrcBPB1afu/LFe9PtuzCQUkIIUhJ9IA+YUpkl4xM4gwvgnIptzBo2EcA5I5Yk1rnzEcNuh2m3xzju4cMIogAfHJwfEXNESgkpZaQsk4C8wxAHjGUv1jEvLMxyo8t9Pbwv4JJRvAdQwM6p+4EoyovP8FmsdygFnKBKckmwxwQMXoovjOOEaUpIMUppdieVjR21sJLbmzvc393h4817HB8/Ii4MLlk37p5FCMORKvEZiRyWGLHEBeMSEHyAp6ziq+iwBFZVWWs6u44dXX+XG7mdaP3pFSZli427+ddElrUvkon315jRBetR0FqHl2ANknZgr0i2JbDSTV36U60cPetXzzC27rnrkUQbEbL/49KH66o6hto41VF2V/RN3SuxNfXS9jMBqa/btkrya35WbRN9ZsJvyB2RgBRYnCiJPMZxhxgjSk7qIlLqQgSLD9gwSIoVu0YpBfOy4Lx8xONTwDDtMO1usd/dYBjU2ufs2AxmLWSQgZQWzGlBinvs97cYhhHeW8YA6bP3hFIcPHt9ziJpiGGZCnSRsdZMd8oEFwmP4ZKRuIh45wOGaUQpGjbDoyi7LWbPeRxub3H/cI8fvt9j+Eu4UPrW7UcljVwKKHs4x4gxYlkWzD4om3RY4qLMdW296pcB1QuyWEnlIM3bpQDDjQGtRFzVoVxkwzRAJdr03X7bMDJjQlu2RttjqYFEPaRnUlecMPtjrAefOsZtR7t7Aa0bHeg08L1+QpuvzxOeFiKzuubmWZ9r/w6Z1E/XLhO+91RzszOZWNJ/SLbjEMyI6ohEQe2EWcFLmluG7kxOXliGlE4nDBozJ24GyBkxJ6S0IKaEJSaUnDBOewxhAKt+xcSQqt4sjBwTTuWIkhmH/R4hjJXdkXNg9gihCLgxIWcBqBr1tVlQ5AjDMAHkwMsZRVmVOKoKUDjnsCwzTqczxnGEp1HGITiMux1u7m6w208YR7MgVSjpauiJnigzA7nAhYKUEs7zXEHcEeH+zQPO8wKUgmEIALsaOkPOGIyrJIM6xtGcK7dOjizPV1+uKpJ7nRSLn1YlJT3FqK4CbeeSbJbWiX4e9Zk+Ab6SFbSww1bk22bvNHAt9W/NO98fg2uMbMuk7AV0Y1P/3h6+Xhdyit3zOS/+62Lvp1o1br2w/SiQunajZ9UFP5P2OQBn2m5Aa5oOtO9TkgIB0zQBEF1LKRLECyJwcYhLBNhhmHZw8Ag+IIQRI0RZngoA50WkYkYKAWHY1aBjWWSuE1eE8ZzPZwDAbmLkLGA5DAHeE4gCDJAKZ/U2tr+LMjTR+Uj8JDAMA4CCpWQkFvcLZ6XbNdvnssw4Hk8SnDyIo+owjdgf9ri52WMYh81YUwMT/cX8rcRIkEDzjEAO3guTGocJj48fpY/aL0nk15TSq4DarzDhFLdq8oMWz8Z1DlSnTWVkRCaTXXqyy7N3LO2CAXVMjO153PVjTBDdMinSfn1Kt9UrgjYMsrKfTgzbruiegV58hyaRtm5du8r19ldlUleG5Ku3664Jlw/5OReGL2tUjSMMaMnw5uawLAtSyQhhwDROmM9z3ZnBCaUwonbFmJTzpPqjAM8A2y5VAE6MQhEuDC28htoObxsmFUZOCdFHEcdyQs4BYRgktcowAERwTvKip7Sokt8LWFk+KV4DlSzQM3JSPyrz8yJhZcKoAna0F8D1HrvdHoebW4zTJKmBYUETrcKxpbwBuwqWOUtIzJIyfCpwKcF5jyUmkCPEzGBiDICWku9YAFBXirEVurIQt4vM9E4X6VE68aeyDZtLG5EGpi7orYt1urT+VdFnK4WtdFIGuM8c04Njf0wdAurqgW6ZVPf3NZbZOLpebL2ajYnK86zBuAJn6fppgOc+gwSvXJs/e3HvEnj6/9r3F8UVUA/50ffvY8eq97mxE1IQyBnDMGKaJizLojhV1F/JukIAE3xhEQOdg9fMCNWVlDNSkinhvHijC+NZ77TOdSl2mQG1wvmUMISgCm4nwBEC5llAzZHU22N2KLkgc6kL26njKjBhQUFOEQTCEAaUMdVCDvN8BhOwmyZRaTmPm5tb3N7eYrfb4XyOQF6/h6pX6t5TKZoOJmWkmFCmEYUJ53lGCAE2zKUUDJ7EsOCcWjfFERbOAVrJmUB1wQjL2oILoUY5FHfJmnUz6OeR+VmJWwXWaKIgs54vuIY4l5/ZwFyddPJPxddenF311ZTybs34Ly62UWs8AxJl0596Fsu39ulzlk378fUIgrQvAqk1TPy49rxyu98Fuxu3r7sJ1X3HHU3u1wn3F7DdZX2M7YJbEYL1u5xZWA85FGRQVUbLOjnFI/b7Pe7ub+BcwA8/vAenjJzF2zqngpwKhnEHP47wg8TcWaGGzBbg7BDKCB8KBholW4Bu+c6Teq27lvu7k/FTTOCc4JPE24Xg4R2wnyYsJJY1UsfMhAxOCakkTekrlj3vHcIwKpAUCfoNAX4YFEQTcD6DGAhB2NI4Tri5vcW4m0D+EZS7l0LNAZQ0BpXVl4khDrMxiWVvnmeczwumSUA+5yiskAMALwUlvNeSgKxB4FRFMkKzvPWftalEde7Ye+03EEkoZ4Bn86YllWuOll1iuyuLtm1kZMXmumnWAVB//KqZT1Gbt9csbStGaM+xmbtM4mQMxgpoL47c6M/WXTSQsgSFa0CurNFG/yommyzAuHjcT7Qvj9370hNf1DprTIdE1xDaQspWgNWfV0dr+zdWf9sOINaadlhVOBf53nsvlq7+ptqcI5xOR5zOJ/zmN7/VXEtHlJxkV08JaYnww4JhmjBMkyx8XbiJxWucnENYIsI4gEvGNO7AVl3FeQT1cBSGb9WIYatf/ZSKBgUnjQMEvNewfCIMIDjKUh4LjJRmtV46TOOo9fY8kFv5d++9io5JE+kBjnYIjrDb7/HmzRvc3t3ih/cfUNIivlY2OOpM6qxkOnebHZtlkfHnP/0Zy7KID1ghkBOgdMwaSqQ5qCDl12EiCQBLNLjey3oFb3Ngbeu9QVp/VjvOWAxWCTNXINOz/Q0g1YNXAPMMsepOqXD4jAtAH2Bs0mzzzG+X6fbuKhF+Um3dXphYmDdft0R5BpL92lN3kS1FvXqflyPIK8Jiut914C+xf923awv5U+4BWxCqx3Q5kdqVdaJfXK8dsrra9m/9bD1XOt1FvY7S6qr3cOgn/LbPTlPh/usf/oDD4Q5v7h/w/uMHUaozgyiggJC5IJWEkCWVsORIkt05l4KSIlL04BhRpgjvRwEvDMgEhGGUggxgEBcV46gqxrNm1GR2mjtK2ABBrGjBe/FBJeEbJWXElFFSQgQAFamc5noynyxHHktZYCLiGAL8KOWu7u4f8M27d/j++/eI54SsGSRYF8bWOl/FOZZ0LgzCx6cnzEsEyCEM4mXPKmJYNon6BtiygrLqzaBWMqq6PIsjhCWIc77enFV7BmVc9u4aA6TV/O3CcBsV25b4Wxv3YMztKpOySXixcgAzNV8DlF6JLnoh7Wq93sVl0H/7cnjo+9Vv9lQ3Hbk31XF7Gfa8juK8AqTWisR++BpCb97PFdC5trC3n5H9vWFPF6DzWVFx3RXuerflVy35D9XXub3sNO1RCqtZv4FXC5Zc9+fx8SN20wF3d3f4+PEjzksEQ7zJHWeACuCAQIPUllPRg3ORisW8IC8L5tMRYdypFXGCIxHNyE26jhRoCpBTbP0oBVyEvbjS0qfUEAVy8CKzYRpHEApSKhJWU3U7Eg6DLMpsHwYEDZcpKSHGBcE7eOdxONzh3Tff4M2f/4zT0wnHp3N7RwBAMnbFCkrCwTkt+pobWMUonvM+EcZJDAjeZaScQT7Bk1cnTwer0+7MNOfae+gXWU1HcoXGbGdPUypvIOICS+hijvcfNQvalYyXVSluPdiqNa6BF63Ot9835P/yMsA6f9PVYqmXbW1A6MS67l/zOO/dCuhiRPv+99d5WXs5SBmlXN3w5bd6jkGtySPWO9czsNzL6Rfine7aq+O77j6zbwFgFIKIKKQLuVbtkOOD9zjPs1b77a676XqpZzCeTkckLri5u4M/n/H+wxPIiaOn84SSHDIRQhjhR7OkFSQwSkooJSKXiJgWxDThgBthDMoqvB8AApJ6cls2AUBAKWUIQLHmQQcBWmwUQGUcPngE1rCWnCDZFOT7EDxy8bAMwjwE5CzGgvP5LN7i04hxGnFzd4u7hxt8//2I03wGxzYwRA6ZxZMeTKbvrjngi6ZyyaWgxAivGR4duZopweXGZDVaRte/WkC799UzIvEzahvQ1lrXzyEy5lcLYuphnSWrOn9uV3sHgtwD4opJ9fP/00yqsbBu5W2Y1HotXG/mMlE3av7U6qWun7VD2wu2Z92Sj0/KsqjHv7S9Lsd5RWyxqLwWEysgPQdQ/fef5I3rl8J1FgHPpYAoXf+NSawmDlB1KM2OYVxeHf0ghQnq5KsTxNVrFeu6TgIm4Ol0xPF8wn53wMObN/jh/XvknECRQCThK4QM5wsGNyAMXsebQMWJ2KRiplR2AVIqcC6DqohXaqyYuTRojk4UIji2fFcCgkG9vEEE7wLEF0JSt8TIiKnCMJzzCGEEWABzCAHJe+QUcT4vCCEgDOI7dXd3h4c3Dxj3/wr64FAWGU0HedZSpHSXKMxL1W1Z7q7zfJKxUUubc16YK4v+MUWJm2E2oLLgGSgYtUXYRDEdF1Xgg4Hi6ltH9Ro3Ml2nRUHpKHcfnfCSZVZDVrBmb2ZEaJvnczzIgKD7G4YNqnLourEtgd76TIAWeFVo0xNsgAiX96fuPpdt7Uaxllza79dyRrV38NL26kIMVRRDYzTUHXQVcK7+3q5b9WzdZ2TqgovrSS9sktikXPlp1NtQ90MpaU/Hu3+lH9SZYTvS6gjeORXzgGYRaoxrBX6QggeCtQSQRy4JHx4/4ub2AW/evcXpeMJ5PqOwwwAnO7OLwmqGAcMwgpyXnFZavcY5D9Yy5qVAANP+p6WrZIzVvwpoYkthsOtM9VxA6qZATvru2CGwOIKWUpCKpBcm1WGBGUgCECEMiMuClDLmeZZ4wrDH7uYGD2/fYn/Yw7m/oKYikcx1yEn7rLupMT/nPN68eYP/+l+/Ry6qTxs8vBd2mPQ4O88TwEkAxnvZUDRNnr5ukj9cr7zXt2RuBZUJqUV3LbW3OafTpGk8SMflyl5KVMfsWSZl85XafL5yIVxjUisxD+Ysq7B54dnaFlZjb8140vRy24emOk7PGaAufdHw7LE/tr3cutcBQJPWP42I27za9aeeWtlN3b2aHqhcS5pV6uutoGDg1iZiMzzbC+huubokVxVsv4Gu78kAiAmk5cNbAvp2ZO3/ZpIwgAzWfE/y94fHjzjc3GF3uIELAz58+CjPSpIYj3IGw8FpHnTnAggMR5qPKshn1T0ATXwoJSOnDNv+g/cI6i0OgpSs0n7nksE5Itg1AQRySKosDyGgLFJtprABiXzOVHQTKcjJin/O2O0OmMYdbm/vcXN7CwpaAFTXROGslWvUKdbEMO/hHeHu/g4xJYCLuA4WCWS2whMg1qz4mht/cNXVwTnuKtiLRzgRWVyyzhQGu7aV9KJhT2YqY7JFXDGEO1BwK1GzvfTGLK7F29kdV+EsFTz61vKr90yq10fZejGouWA9HWisHTM7fRgxLrzmu2MM0FvPn1/wVQS9YGXPH/uS9gqQ6gCn+6iOHy5Trj9rrdt8Vj/tmRBfbi5iyeozoq+V25VGqzixZjzdLFwxrX5A7eWsP7VFvp4E1Ppc/a2ug6KRe+mTw8ePH/Hhwwe8ffsW3377Lb7/4b2EuWiqElZqSc6L06eX3Oh+kCyWPgRJmVI3aXmgUkzs03HNkg3BKwiQo7rYSilSiLQ4uLyITsx7ePYoXCRj5iCMJyUr9gDxmA8BhBFBwfLx8SPmeUFKCeO4w+3NPb559ys8PPwr4vl7xCVrqpmiHsr2kqkG/I7juAKHPqMos+aez/o9AawZG2T4A4gZ3olTo0hCDaiYLOJP3q7p4dAdy/qe+5Qopu9Zxd/VacqGVXXe6EnomdT1jY+N3NdxeG5ZNwlhy25ayJHdYWt8av8aQeiP7p7tAnj0GXCNSXX92vbVzln16se3V/hJbQMgua1RAJcy7RaM2oelMrH+e6q7Qn/29qo1VLYb3P7lFAUCgwbU37orkf3dtE82UXq/E/uKIcyOnJSi4gqOzfRqz8gsVy31J6Ew6fNxzd1USsGf/vxn3Nzd4eHdO7x//wGn00mCmCmAkEEs+Z2C8xLtT8KwnNc0CebNzhJQjMJa9Vtr7jGQS4JL6tSpITkwDsoAJyCBkX3UYp8ejkWcouBBJYBzwnk5Sppj7zGMHuM4YneYwLgBoyCmiJgjCnYYdwc8PLzF27ff4PH9CTmdBKQAFCpwRFK/jcR1QgKZGefzExwYqViKGRm3woSUGKlE+FwQ2J6ggJEh1ZydiLC6cskpO3VUFw4XgNhp4j5RfssGxBXcLB7QgsrbfEH7u1+Gldl0QNABVf3wmm6nXxTPoxR6r3a5n7qFFKrfkQKp4Znyxtovxw597+umTQzTXlY2x2a9NrFu/czd41a21fCAASrPPs6XtC8Pi1mxkev07YIJGaitWFK3wNENCEOrT7RWdFrWC615qP7Qyiyr72mFOw2k2lCuvZLRIuRhL1XYDFNA1akw19/tBTEkEb0mR+lf7eqHVXf5+HREYYc3797h9PSE4+kEpoBhN8GB4HR+e+dFQa1FG+w5ilrjSk7yzAqaXEotsU6OkIsXVwHv4L0xA5bEeMqWYowIPgAQMc2pZW8oATE6nE9nlJIxJvFYH0cJLi5gPD4+IibJg2U+U/f395imCafjGVauyoZeLJQNGIgIHz98QFzi6j2Z60BKCQUM53IdVxPFQCRlwsj8upSJriWV+jL7qVunCXXEpn9l3C3KFZuyTbQT2/STC/60ufDWT+o55XS9Wf3Z1lqd0dzNK30YkR6vXfNz0MGbY3oXoGcKJ/CVtc/XeNaXt5crzjcpT3j7tlfUUI/hzSTRB5KxNEtMC9gt9ZqaVqOsB7UCAvgCADcHAp2+qYHS9jDXTbzNLDXriX5dQAA7ZAMp6iqyKLoVbgtIY/31b9OP9deWv7wjPB2PmFPEN2+/QZh2+Ph4RCFgN3mtYuIQvMcQBgy6GEUEkmouaYmib7J3wawe3MK0KAOggkhA8Q4cfDXskPanlIySCSlm0WUNUsyUtMyWFE5wyCliOc/w5OBJyrWPYcDgg7hApIwQoAVEbzDtJpD3bVF3wGOT2fRS87IgxigZGYp8XzSQmuBkk/IMuCzB196DiuiliLhWiq7PBWFDJq4LueoWorkvKZoVB7ia3aCbzxbV0IFPUXAxD+wKNOb3ZVr1nnZ0M693gFxTd2trce6TrQMKE4HJMskofjKtgaOOAre/za3iOsIoI6PPrL0XtWfi/55pP67uXo/uevP+CXkDXL1V0D42sa0Xn7aD2Z9f77NNmboV0MlhFSJBl3J/79ypa7tzeqMaU6evR3yRnKv15arI4RgokieIzWzNUN2YMSzpQ39PY2vkHVLO+Nc//gnvvv0Gt3d3OJ7PGEdIFgKQhrY4UaQTABRRuCujSzHVggw2VvYoJWcUghbgFG8h5zqwWA1cQSpSdNQHU9Q7DMOAaRyR4ozCBSlGxCUAg1x/HAakRLW4wzTucXd7j7v7e3z/wweU3HglszA9EKE4ycXlvQcXrlY8gAFHSC4Kg4JX0sBAzqDk4XwBkbhv2KR3mvKEWABErKKy4RQ933d502vSPBLrZlHRkxw3xnoxb2y+cJt3vaL8QmF+bVVzJ8Z9Rd6hS7IyRPu47+qz5zKeR0b+/CGv6ORrgO7LA4y5vaye/q4Hpq/VJgdXu2AFp0aVV4xKr8Ur7KF6r4sds/5lRPgae1ozK+r73T2LgQl3xzdlOHVuDATWHZ6poDgCc2/9c1r0RCYkKzk0sbIKBvX6jO9/+AFv3r7D7e2tWtZYGIPzaBk4pf9On5Z07FolG4NVY4fKVgsLSBFAFKrIaiKXvSfmDDj1v2IWxkSEYRjgfQDHKG4BMcF8gYL3erxkSwhhwJs37/Dm4QH/Nv0bliWuatvJ08q5kupGskKUUrBEyfPO1MJ3vFcFuKYWdlzkZ9F4sczInOGcPo+ClHNcw2fs5rVQeKd3IiIUJ0DjVcxm14fKrKdRrQBP7XwwN9BbtaYr6yZZm3lbRIERMe5qUTYC0KdwwWYlfGrtr5+D1nMfqPnfP8Wk+kwRP6a9Bui+gElRTS3SrFrKFy7EvX4AWxiJnVF/cudrUc9dvzsBNmNGjGpLNDrPFptmQGUg08EYrcGLr22UGxlcuivH55Kt0pKOsuZLItnrDZRZgyxZz1fHaek1cy0/XhWygOpn5Pk+fnzC/cMD9odb+dY8xK3QKCT5Z4HF1/kW+KxOnc1FobkpCF5JupVcslSdUXnATM21ECg5YLQhJjBnZVRB0iWXAs4ZxAMIhKTK+xjNMjji7u4Ob9++xd3tLd6//wDmolkQ2sZkb/fm5gaPT49S4l3T0JhejUqpVnJ5sw6ssYoMQmLRWVmBihBI8lppwRqnoTPyrG2BF+5M98zifOYIRd+rK6Lo36LOappX3Y/Oq8KNIet1mciid1CpTrfVXtepX3NTsAOLPod5hV2KhpeX5PWnbaes367BcItECvDcibY/ov0kLgjXFIqNwNsuvAWpjfhXgybbeQZEz/W5+jrBkF5fL68nB/XH6q5ZWYY6NzIEDOy+VXwAKmKJvqkDS5KJl1jEK9ttuB67QVPxIGyMsablILH42WZVJ3L9B4B4fxMI59OCYcy4e3iDYRjAjkDe1XG2sfbegWgAkeihEnOt41eS5EgXnBWvMK9J77L6SDnvq57Irl1KsxpO4wjyXjJlliLe5s5Jjb+cMGTJ4CmgVcDEyHnBOARM44R3b9/h3dt3OJ1O6rjpugXV7nl7e4s//OEPiClhHIf6DjILazJxqpsY8hwpS/ksJUrOORQ4DDTA+VZGvtTZYHdWwFJGa3GNYKBUixm3V9ttVuh/vbaW2ZhQO463/e8PxuXaMSaFytqxHgM2Nnbpj7UKm6ndXK/F5l5x2Z/rS1Hu97zv10/XXgxSuWdF3JzIKtgwYLlmrFVnOMAQoVnNlF4YE1s/uJlC9X51pPsX1Pen3xEcasyV6YBI9UvUdpwKevY86D9HNykZrNVfiuYuZ7Zng+6UXVhGpdEFIAYVCUORSdpRaZ3AldlVxat4nLshiJWwACFMIDBiVN0SsaRRhwh9nhjwATxNWIiwlBk5y4TKuWjMmbG4pE6ZTqoKx6QWv9BVnpFwlZITHBe4aQJzQsni5uDgkEtCiRELAUMYtG6gWOayj2A9bjftcDjsMQwe4xjw+OjEMdNeo80t53A8nVoyPx2iUorkiqcEHwJMBweWzA3s2hyzEmHMQM5i1SwkVNaTE+ILFXttW1NgY1DNz+UZIrp2Sm9h4twYub4uWv1kVdR3xxOBqWj+dHV7AKPLWqUAegkyW4uzjJmBk44Q0Uqt0jzB3QZASzen2/VtCW3Q+BoVw+U6vd4+x7ReC3JfIO61CSYgZYjcgMuEGIZNRN0WjVXUz5qFrwEdusXs673WO5mdA1SxDKivuib2MuyqINVdRwGiZbxsliagsTYISVCfHXG2zFUJzJIqlYGqMLetz7K7wRzctruYXp5qJ0Hk4N0Ap9a0cZxQCmOJC3bjpM6aWUAKAFCgCVpAjjSbZRFWw+JvBFg2USlhRRAFeuGAQfsm12XNc44qLjJrhRwi8eiudeTsMEmfLMeKI6kpvqdxwRBGjNOE/V7KcBGJAr7kBd3LAbNUjhEFukOfhdT6kbPkPA/e15g/yZnlwc53GR6kj8ImHRypzpAzCA4tN3l7F8wFXMVp6ZPbLrTKupSlV1Gd63XsZ+/8S4KCyIYRWlWoHVcpYLsXN/ffFZPCJUOTubZCse6gdatrsWdfClAX4TFXcUSA9xkE6+7zdZnWK5iUvoTqNGYil4BNsbHiPtxlQyn7dC9V9OvBpgO81d6CCmz2+mrwr7GRurvJZKum7p7I2bGu6XaMAZklsCVqs0UkOidTfhabOKpAlEh+VsDTPpNXdlY0HrDAYqYkvqzbnUHSX/WD8mGEDwHDMMF5D+cdckpIIEyTVJKJMcFBQ0HsKTRrgSMNXSksHudF3BiWJSHHRe5oehlmNcsTSiKgJLjg9X3J9VIWRije3FCRgwFSF4cCJBLgcwSk+YyUIqZxB69ge3Nzh8PhFt4D79+PiEsU51h9wdM4IS5aVJW8MAsmlMzITm2ozHCcwZTgdZy9l3fmIMYFFCcACwYcIcYMx1q5CxBFOolbRZuH8l8hRobo3YKzzbbNQVUWtM91PpJtMAb4sP2RKthQXRx6LZLPWy8IWqq6kZoKJNgwKarrrPlG0OoYnQj6h2709Ukst73pclHnpoHis41J112ddY1vdIdtq9r82PZ6nZQqv1unuL5o22kru+ioaX3Z+ouqc2Gj0sBCBrCmS2m3acd3rIqB6l1s7Md2qMao9DpqeWn6oOYUWcmv4/qMcopaAa1OneYGKkZ9yYC2A7fWM5113SRaKSwFoLxWkBk0sHgaJ0zTTsJfvIcnp5VpGNOkqX9zUs9w2wrFy5whmQPGUZ7He0KOCcxZEujlJHoxTaHi6mYujCLwoNVh2jvLKQHeo5ZY4NziBPXB3BjUexw4n844jk8YxwkhBLx99xbvfniLUhJub29wOkUsS6qzYhwnLHGpi5JLEfahOiwwVFwiICV1lGWU4hEYoEDCUKzqciEJLHYFTpXhUtfPwxOtREoQVcU6SJT1SdeqJAVE/Q4KPkStlqI103UK2ejDblQHp/I5sYCk141S1ojF6bV5camT6vWGa/KkVKj1s5uFz7cVHXvRGUYg+itY/9ZW+E6U/ArtxSCVVMJRciEd7OCzUUnbc9TdQJ+/7vkKcFWKpq5cE1AZVPUmrseh0SGbQsQ1hIF7samrRd8mY2OBRtltEsmM6F6YMSgt9lnV76rvcHVHbeBUxcyVcYD1cmIdQ51sdoT4PYVhwn5/wDCMCEPAOEwYhxE+SOUXBwI7RkoRzBlhCCBNnieLw0BKXQr0/QiblAUxDANy8AJSOUt4TsrwTuLxiEjFPMCNkntK0VmePhYkHZ9cEuISEZe5inDeS57z3TThdDrj8eMH7MY99odb3Bxu8O0332KJZ7x//wPev3/EMkdkzcZ5CIPkOFdrcSniTsBw4BQlltCxiHa2lxPXMKCaYZQI0d4LEdg5CSFyDg6iU0xejAeS/cHVdyFls9pGZzpHJhOJuM0jXPgZw/ZDyL5ViYzEDzZpgry4N0A39eomQrrhsWyeZbOBt1llG0q/ALc0CjWttqsl51dyibZeiumMBJ9s7V4ruxivD3lO/f4l7eXinukjrGdsavOu0wzYCqEuFWup3twNdY2piF5Tvm+ZMtSMu009QdTdr+1S7NtOCGogV9VK1PyoqDsOK7AinSB164IptWPOIOfleMurXUo3+Qw0dQevYMoth5HpQsxyBKsQPGG3P+D29hajinjDMGIIQ31WKlnGpxBKSUipYBoGOMdIy9I2DlEKovk/CdhkXcB+GERXs0j2gpykmIT3Ug+QQ6kpf8XCptczyOVOh5USzuez7vgjvJfqMdM0YRwCnp5OeB9+QBhGhGHA/cNbzMsRh5ubOjcyS5jTqPoq2HMAyJxRcoErDlxE7HO+IIARmmZA/9Ed1HtQ1o3HyX+khTidc+KdXgjZZRCpeO28bqbiwMmealJDx/196ixvM9DmaAfmKxsirbztoKtXnYFVsY+mVW26ygKw6sdM/aAIuPZUB/rO2dTt/ai2FWCut6aA+Dyj6tZ89+sKJqsi/+uwqReDlKVdqSWJjCV1g8CARrk7kNMJbnoaM/Gr4s1ErjVoKftSRsJE6xfcP3TdulrkuxyvQEXt5brufMFLneXGvnqG07T3AJzEteWCcZxgehxUhSNDcoFISXMRBU2proxS8Wub60dCXSRt783hBjf7G4yj6qHUDaGOqisQnJLJmfOCiIz9fieMaomKtaZr6t6HbSYkvkPQAp5EESlG5CjMKvuMUgJ8kCR8zEUTyum7tpdFEMV8ych5UXaXMGi2Th8GTNMOp3PE0/EJ4+NH3N09YLe/AROraFXAJFa9XFir5gSgs5La5CjqiCqMidv/uKAUL2I4VOGfxeFVqjlLULZjB5AXR02Ik6ojV73O2TMGGsBUkIuk1qlzXudzG81us6uvR0BKWKt+p9WkCbKnOe/Q5AUdUTbW1EhaI2sGfHZf24D0c2pD1PpmJIHqa4LO563rBrrveqTZuilcb5ffbz9pY7MmMfXGr2yvUJxvbsx952yntW4wTIPYENp+bnI42c7fsSRWcFsB1Qqw0IFUs9AJde5j9ro3QoD4S9UjmztEvXD3wpT9Fc5g56pTp7EzLga0Nv2MJTVwMXP16ibGKsmBvJQ4H8OEcZi0SouHI78ZWyBSAjJqwHDJCcv5LJ7aAyPFWO9i20jp2Ap0DJ2TcBKXmz7JsjOImqpoHF/COAY451GodBsS60aVwSVjWWaAC4IX0eqwv8E07jDtFjw9HfH+ww/ww4T9/gYhj9gdDpKWxR1RuCBmqUGYSkYGRFdj4T1UOTGc5qGyhcssVXVSbnGBmuhcGZIo37kw4IAcc13EUhJMwUSrz8hzriZ1P/z1F4PtbiXqn6LfY2U/DkXFNoYrGcHJxlS4wDHBPE1FXeEg+cr0HkSyyaOaVvR+Ba6+4H7Ttk5aMhpjeQaCHfvrss22/64B1HNgwtf/7NZQU1ttrsF0+dkL2itASssoaQcuk2Y2S5oorl03BD0R7iwr9hU6PVTnI2UmXwZkwm2UgqSe2KZXqropS8dhop9r5xnotdAW60lzpygk1kqw7PYMAzQFIe2XgKPYll0BisbG1V3KlKLSCWFaisxEhOAH7KYJ49iU5hKfp6Ily0gzMxwCMgoKcgVM1tJag/cgLfTZ/NnqCKNnAAZYvBov6WvJGZELXCFwEbcF77161Os7geh8YlyQ0oK0zEBNpkfgAgzjhCEMCMFLia/jETc3bzBOezy8eYfdfg9H77WKMcOHEcuSqphq1W5sbIkIgRjsAa9W01Kk/5TNaVfBh4CsYCysgoAsITSFGSAPFEkR49R1IZcC6qIZrjbCKivHSjlMOj9thmjEgflClSxjY2XrzfoNKgpSOqfrBibziFy3pfPWabMJirXbKnaaV0+nhzd5oz0j84aN6ee2OD7ZLpBpM2zcfUXrz7F2x3lJe3W1GHsRLeCiNWcj0gFUS56gu0JlGz27Ql14PcuyFyY/jDUpmJEsNMnb3eukfPfW2zVaDwzQdTfr+mC7QKXBZCBl/bLr2MuETpysE1WzN1Saa/43disDOlkkwzDB+0F0QsHL7861wFYV1YomeBNbAWnohRgOkMQVcAwB53SGAyOjATB1dwWadcapxYscVZrMorUWdxMWka54OYYNeAEAmpu8sFS2QUaOCXFe8IQnhHkGkwdYRNH5PEvxUT/gcLjH/nALH4IqyBneByynE0rKAHHL2V5DegiZsr4/SV9cnPgyeXhwyiherZ1BYv045yo6knrEEySAuFRdm+Sqogxk16bR9XZtYbHqU9Vj3fJSFdcCukleRikF3ovllUhjEZXpkObWItuYVGdKbCxI5ptznd4UtpbaWyZVPzTjU8vJZ5+RMv5ezKzPVz/7lG3uEqC2MHRxBnfn6O+l5rD6fHs5k9r8XaAhHv1nHTRX/bqBylYnYwu+MpvmA9Wu0fydVpkUAZBjSH4nh9JVPkFvCpYPL4az6fE391OZv7IG3dm99yuz74rNcncPMTvJXiH0SzNMFglpMTqPILqb3YRpGmvgbvCyu3t9nlI0s4GKWBLW4uS6uguXnJFiQRgGDMOAZZGSVrwq776eWkQE7zwwMJgHpLKsTMsASwn2JGmCnYbkmN6FFaTa3NNdmiWtSs4ZcBpKo3olKX01Yn+4wcPDWwzjf9OsDAXjOOLjDz8gxig6q6rra9yGuCAjyzvzogCHc3DMWpxCUxyzW707BiSLpzIHCiIR5Fzg1dUBzFrb0a1W3FrnahvPuhkHZw1OJog1j4RaGw0CsbC/GmysKo6a5rgPF9JNVn50bpaFNHyT1utCWT53IGAQZVtsWc32VQRrZVhrgOlnDF98uv189fHqwN5quJmNLyRUr85xLpkjAZhlrfu6WfvaF6VRonXH7E9T7DGhJcS27+1NdXK2KcehLxltC2Q1IzenysZ9+kui+6zPIVjldwIyyW5eIE6Dhc2JT9mNPYhS86rQJbPumQGAa+FOs7r4EDCO4o09jKNmxDRx0AJfGeAElAgqEY4TpDhdqdOv6K4ooJKqQ2iZI3IRNgFNKZyLJsBj89sBvA8oLiO7JMdq6Li8TEYpkjbYB8LgBlCv7wOkVmAYJFzFifGAS5bNRRd+KQUxRixxhhs99ocD3rx7h8PNAe77HwBeMHiH83wWsOEuIpRkM3Jw6hALgJ0mArRFCgAZOZNWPS7IKVbfOWkm+nHNmV5yRo5R5ksQIAOXzoeKrqo0nmuWJbOqBTLgnKXE4RqUz5DMDRYBYc60QNvHTXHunIPvlk6RR5VzdJOWd8n1vLqWLthQX2JEXRhsQ7dVsgGNul66za5d9HJ0NO7omVEzYNJSaStA/XR7uXVv88jVI/wCOdd/U/23W7gQXQFRf4HOeKu0hBSQDHjsKKPwRO2cPpNmu2MHTrVf3V9EWDF8VvrOLTZOZHfVPW3bSkcAGGobeDtQjfq3CeS8E4AaJ4zTiHEcMI4Bwet+VySYtuQC5ohSFgkjSYuIlUUWRLFSUGrKySkh5gXjMMJ5BzY3gyyBwFkzF9C6+1LdmDSTqLI1mfg6ekWCeB050NBZtSD+VeM4wSsDtPAYZvGyTyljPp/h/BPOxyPCOGG32+P+4QG3tzdV7PDeYZ6l3h6zxjxqvwisBKcXpa8102cxgKQe3FTLdrG6IhQtP19KUdYJMEbV/Zgd2DaUrf6kwAwl/X31JJsudW9thTLkS/PVXmXBqEBbVjPM2dwyJtsNfDEQNL2iGqnkfWr+LDatWfVQxGYltCm9+VjYYFu53P0E8zUc6o+4HJvV31x/vLS9oqSV7343wvhc69C5ftYi+NsXPUA18YzZaegC1xdRB81ZbBd16U7IWPXqb9eLYteeaaM8NBneqpiwggUBEMtMR3ZtZ7Zjr3wOQHRHDHgHBGVQ0zhhGAbsxx3GYYAnjb8DAC66yy9Aiih5Qc5noETVGblup1SqyMYSgGWe4UIA1GrFdfFKoDHYnFRLTZLHgBZmKGDfRA8PYbeFJU2NZ1+BC4XBRfypvA9glvi6lIqAFgHLfMbTk1TDubk7Yrq9k2woTrIveCL193I4n2dd1DqJHbddnJWl83qPsTF3tpExUFLW/F6ykUnwcAE0nY04bBbJFsoshUpdgieFHxbrqS3Gnn3Yb5ZlYzOZ6mom83Ei1XlRJ35qVgYrimFiGBSoLGRHWJMYSVxBSzeja0GyPajjrkVSsPp62TxlRtttZcSaP6DhfQMXU72sHKHJmJfKSmQbxXb7vy4OQ/tVxT0yve/VQ6+2l4NUlzNZno1Wfar6GlYA277EutW47s+eCNquItzJVT0TOpBqlhyA6nft/M0A9/e9/lTrHhrA6N8SimI7mvaVu0nwTLM0GDYpCIRhEJ+oaZowjhOmccJhv8cwSJ07UKn+RyWekecZeTkjLzNSPgPIyio9iMSnyPsAS8ovtyxIKQIlYZpGlFJwPC51kljYSNGwFs5SUl0mt0NeFqBAlPiq9HXeiwMka3Vk7zorXJHsnd5LDF+SRHisLGiJEcs8w4cz5uUkmRK4YNofcHd/j3EapRCod0g5iXmeLCmfvTuDihY3t3qDzDBVMZcM8dPQIzX9MSPXMmBOqWJwHtXsz42FEUFAmHqjj+0LVKdtm3cyj9qU6DZcxVnBWxF9G8h4ZYp9K10GVrlKUWudWW0JkHL37FVf6GqqYCIIG+3nYGZbGtJ/7oHKerpe26TzvR3b2NjaUbNfByIJXG09++J6pevHXmkvV5xr+Mp6UHs/KJ1MdHlUHaUa04QLEGuVO1Tv4frjFMDQRMT+Ohd36+/BWyiqPbpoVdHKog84nU7Y7XYrYHoOoGRut723dVv8dsQbe1eBahp3CGEEUW+LKeCyIMcz0nzCcnrEMp+Q0gyHrM86IHjJGx6GAT54WYhF2JAltXOOsN9NWOaTBBqTjqnKG5IlQcGKW/2eZFaxIagrhO7GxaFkRknFdilIReJGInKR1MNUhLlwAUoumOcZp+Mj5uUMZmB/uMHdwwP2N3ss8QnQc4v6xTqgWb5kmUPCp0TkK8yqvpNF76yqdA8bRCrySXiN9wzPDB8ACh5OmScxkNWQSFQ0tY78V5ixdnuhliihnzfdXJP5JrBpOCGsiBtbAsE5q5SDCmYgGWqnbiBk9yOgqj+INHwpK9tVoGqmwhVU9F4NNh9bXnbpvdM/+g3eVrQxNDmyFx23gPQ8GbA1tR6w59fStr1cca65hky5RjDv7caGmjP+FoBgJ6G+7q4SQLNQAKi6gLZbUT/K6NlSNzRENY6v3teuv+rM84+4HcxhGBAshxFMPFzroa4PtO5EJKxkmgZMk8S1TeOoADWsLDWAsKB0PiOfT0jzEcvpiHk+IcUZ4KjYPSKEhDAMKDnCDUH9g3zNyABmzKcTdrsD3jzc4y9/+R7MTYwQMHbiIwQB11yKZFxgrtlHbaEZe2UWa2IVqfWnkBERk8IQ9MWoQYOBkhJSjCgpglGwP9zg5u4eN/d3OC8ZcKipivv9mU0AN5qgb9KMGCgSj5dLQTO8A5bBtV6HCJnFxyo4h8BAYYmhyyKnAkSrAisy91wV7WQKdlvyalK1NWBMo7oPuE6HpR1yYAF89bw31gUSPV/QsKEqWpNGcZg3vTpQEUrdcGrojxkHdM0wUy3rZWu3ri3WtUptLrcg5vVcbymMeP0w/Zx/Bqiaz6DNl2sg93x7MUgR1fyt2qVOiarWLJtGZWMmoNVvVBF/pWui9qJNOW6fVytO2yyuiHXdHahzrHsFrbQns/Ech6ATkipdqHvJCpy4/ldN3QQN7A0YRwGpcRoxDiOGIag/FDRvkSi547wgnRfkeUE8nxHnGcs8Y56fwCXJDkwR45CRxwl+HOBLkLASJ9mhSlUeA6fjEw43B9zf3+P7778HnIdjgvOypC0rZxs7VcRWUaZNLJlcBTGWOsFNNChaVCH4ABo8lpTEebIAIA8fJLMCq06MiLDf73F7e4vjcdFkfEYnum1u9W5bj9o/CrzMoEJgdZyV2Ds1vhPA5OCzWh01o6gLouCnbHNSYa663Omm533Xl6YHXYVs6dxsYlG3K/cKGJ3ElrnFFAsEWLQLcpLF6x2tHp9srLloih7LVMGVURbmlbhMpKCirhCVVXVvVvrT1qLN6q3OVVfGZ9jP9e9W4TfcpYV5YXtFIQYLg5RmL7E9tiI71nPrua5cAyi5hgGY7eDtHgZWW2Cq1+v/Brq/14Pb79YN2fVIlh1zHEeAC2LM3U6w2VVMnwF1SuC2kAgOPkhpqHGUwNtpmjCGoZaDkolZUHJCXmbwEsExIc0L4rJgmc+YT0eczycUTnAgjGECOu9mEFTZ6sTqx+KMaVa2x8cnHA4H3N3d4vHDo4ik7LRsOzRRXlYlu5Pslq4ZOUhpgyOHVCQ5XdVxOXXPAKTasup5CAXneZbyVOOI25s77Hd7ccXIwhAO+wPevXmH+SQpW2rkQJ+upHt/oiZi6ctmVglG2jsR8pHtTVQEENN/zAWYFxSyStBeggEyQIXRp20hklxcznsEgqZgkbGzoPl1sj1zP4HOACho2qxoCYrkxSnjNkYDFm1HysiETjcl78GRlCIrKto7kuIRpscSCbfAKuTIo0scK5GtTXUuriSDtCvc1qRzzzKpiqafaVfBzJTvlVW9rL2cSXUJ540i1mlkylv9u6fNtpAurvcM0KyhZNu6h7P7tzm9OU5/a1uvbsDd4BlDgu0S8lvVSVSFi1lmOkGkUmf7TsHPXraTSifjMGIahUGNw4jQJY8DQbMaLFrKPCPniJQl+HdZZqQUpQBElnS2xKlVRIZasLyHaCeosoySC1IqAEkA8c3NDaZxwsf5EaVkUb46D4RBjlefJvMTInCdxOQ8SAsiFBbWVzQmLiYglALvB4QAMWOSwzwvOJ5mYY7qcjGNE5zzCN5LKfZvvkWMBcEP8N4he9OXNBaxFa3rnqyspH8P622qsyKRuDWkIsYHgQtCGEYMQZaAWDVVPDP9khMnTF80YEsr9gijNahpEkC/rQItKL/f4OozqULcFCSum3sS0UOqaDcXBPGUp2JZHTTtsKandl6rOJNmUa2AagxJ2C4ht02gbvhGAJykdlGxe8WudL4y8YU/Vbeg6tgTgGs49SqznrZX6aQAiDLSFoO+KFedwlpXX5KsoZ9ovV+G/SRgDWZO6asCFFM3MZ8duUYse6ZzMVgEMJQ1kZqqc4Zof/vJ1l8318Wh6Y1kMoMRiDCGAbtxj2ncYwyDAFRN9iSWtlwzEShIqSJb/tYCpAWwTIrOkVQidupZba4FzvRmwji45OqQcz6eMZ8WPDzcw3vCvES4oCLnEOAJCAws8VzFMctmEYZBwjFQgOhRwDhjRkkJxEUsoM6BfECGMDhHDs4PWFLBshzhhxG7QwSRAxfxuRrHCbe3D9jtPqJk0oXYLXplgtV7H5b91bXUPm2/AqqpvwMzW1hMSIWRY4YjKRWfMhASo4ysb0yZIAvrIiIgiQLbcZGkn17CWUyqoLopyPRzpgvjxqwqtavPJpEApdofhK2UuikwKANB3ScKq3eWk/fqLDCZPepocRGPfEcY1IpJIDgvfn81FbKF7VRRUlZZhrlxFFXrNzrQ5CeFYUs9ernMVmtJbndJCEQbCjzn+Hmtvbrunu/8Nfqf2+bopX1ou+Zz17rW2s7JMH1B/WYDQFv9USNEnaJQE8alnDAMA+b5jJyb2XhFe9ut2mcmjuhzhBDUiicOmz6obqOCSEFaFuQ4I0Wp3FuiMKi0yE9jN/ZMBEkNPE4TvBvFP2b17Nqfep6ATc4FKS0AGLd3tzjPZ8S4AMOAoLnRpeMFWR0crXzVMA5VlMAkYrALRxyPR6S0gJOOke7GtuGEMCJnxunpCYUZ426Ht98siHFGXGbknDTzww7necE4DphPJ8AyVTwzFWybqSRYJz4pGFVRQoGNWcDMxtGBJDdaQn2/gIirloCRdD4AzSHTBWGz7FljKLVDJO+/d9ysPkXcWAk64CplzQrtn14IKkVnt+74UmJegRo2zk1/CmZwBlJJVerxLO/VdWuVnLmxqMRjYaI14V6TBvr5vlYPXlvZRlxWD3V5TifqfXXrXpXV+x7rbGo5jZvG3sQ8BpolxLYZAOuPzJ/oM40bK+r9l6qyGh3Frqdw7delGwE1HYIqJcGMMDgcz0eIZoOaTqruzNpzm4D9M+pYea/Vf736vRjrtHslZU9Zco+n+Yy8LMgq9hlA5ZxrIjopFtpECu+clD/3rr4LguzSjsWiVVKuC+h4PGGadtjvD1g+vpdYuSCOlSF4ABMypH6g7MQekuVUAMuHgHHH8INMm+MJiCljyRmBCwLJeBZ1PSglI7MkTMw543Q6IowDxnEQz/sw4M39G/zpT3/BNEx4Io9qHTP9yJX3WZ150d6FvFP14YLW5NN3Ir6nsggzGJwiQBIJCOfgS8bIRbKC6SIyx0sAIBO3vdQrpELw5g6husimszFdUBMBbdaWrh6i9LOLzCPA4vqcTubSjYNmtZY0L91zk92MAE/VGCjrQh2EWUOuSLO8gqFB466FkZludVOM0iDls/yhbg7WgbYWTSLqGW5f4eZz7cUg1RiU6j60+yafy0M0r/RSf65ZTIOiBhhbT9f2vf534ZzC3flGvbfYbedC8WQLUN0x3fdSxKBUi1W/Y2/Po+56K+W/TiyqxxadMCreloyUFkleF89YlhOWZUZZIjhHpJgkta4u7vl81us6BD9gGEfsxwnjNCGMgzhYQnQKzAzyJLQ8s8btpbpL/vDD93j3zTvEnHB8ekSMBQMRxmFACANKCFoJWS1GuieR8wjjKEnjBsmvxY6QSulq6pEKDZJuBQCmacDNzQEhBDw9PSGVhBAc7h8eME4jHu7f4Pu//AAicbSUclfN2mT6IdMVObJ85WbZK22jMV0SUa3cw7AAZNW7KVPwLkiA8ZW507zD5b2LJ32CFfBgZmSLF+3mRbVCV9xZh9UwrwGtWQP1FGNlOm+ciomsMYFUuBYrNR8rMWCxKs2bDooAsFfFP0kWWWIG2NwXRM8FdZwWJlWqCNo/0+W6eU7qsZQ+m7VWRb3+Op93irb2ChcEpYx6A6GnZpWhi07bAm6OlSoS8ebFKkiQUaGLq9j9GiwCbcczOGwZGKiCQbfVrtvFR7afSTqP8/ks/jN1t0N3LX7mIrbDo1lb9Lqk2xtrqEqKi4h4yxlxPiOnBSWJwrzEKEp01hJZGpxccgGRlCAXT3gnmRVIwzRYdQDc9Uf3egcglSLlzL3H+XzE/f0tluWM5XwWUZRGEfu80yybGhtoxNmTOEFqcrtdYYuyAbN6m6eEadqBcgFsYZD4iuWc8PHje+Syx3x3AD3ci96FCqYxYAxe4hfZdFMmpvi6GG1BBR9gGQuKirNeQ20kvEfnhma0yEWS6/WgABL/yszQ+oZcRXubX+bL5AjgKNnTRZ/Eyjxpsy66qaGB3+bJ3ouE/fztWb2tAwZrokHGygEa3TpUwPbOVYueBV6ureUyh4hEF5R13Tp2VSw03ydhUrnFxa7WNq/uzfUZ+7WwBjRTg1T102bt/2QgVXNGobGfNib9AImFQRSCmx2jb5VPXqOUW/ayPaB74MId0D3/8OsrGNjq/Zkkb7fXYNb6rLy6fQ+dck+jtfL8kjPcS7Q6GOCCnDWYNWekRfQyy3xGWmaUJFVcuGTkFGXhq4fxME4YkyjYi+50wstk4me1AIGcKu+Lep4350bo5Cs5IXLG0xNjf7PH27dv8P2f/1KLgnrvQZ7AxWEYx/puGSwMyktMJeuAuRCwOxzADJxOJ5xOJ4AZIQxwHhhHj8enGTEucLNDLhnTFPRdSbziOHk8PNzh22+/wfH4hGWWMJ7CnXWRlLE7S1TXFNSFJd2KGwY4F5A5rha+OKrqLl6ZFur55sGec0Z2WbN2GvhqznciFI6iW+smsvehsQaYpzxv1sB1BtV+toVdF799V6WMDTjYWRo/yY41XTfXyA0TrxwD7GSTdMzwbBEdLJunXlckIhUxiyVIkF+YL0nIeg21tSS/KoO0P8vWCdr8pL46SPUR4mtEX/EdqpgDrgzFvm/E+nkUfY6ptM+Y2/17HYYp0w0yqjS3pZ/1TTddFuei4Ral5U6qdzS2olyOUOn3tXHy3msFFmFTxUzfpSBrNsu4zIjzjBwXlKwe2TlKFZcqrzuQGzBNQPKDKLtJdAmsP2UBCXRZulFbEJbwX3ZBVHEi5YTvv/8e337zDR7u73E6n6W/jjD4seaBgrJBkRlcjToomk7FMjYc9jsE7/D4+Ijj0xOGaQRr+XVHJBkallnAX8HTAygl42Y3oby5w6++fYc//fHfkGLs1Rr2JiWXFmkVG8iGZi4iPgj7M6fEugQUfIqdIMTEXr2Ah1pQxf0jqRG7Axmw6KGIqghO5t8EUrFTZ7sRuO5sYWifWJA9s+g/L/WyMH+q5hTZREmZV7KJMK0D8s066DQmkb2Bkm7CGk4FZ4Dd1pV5/5Nt/pbKuwerXrLoX9iVR12vQd1qvzaTqmYAMlm2ARTp5z2LqUDFVxhJ6/nqd9uTPqukq6e0QaovrNtFCe55gFo9mbyAXLIou7uJp0YyyQBqeaLsDpbM3wJTyWEIQZLYuS5lBsvCzCkhLRFRrXc59ylzi9axs1cooE5EGKZJq66MYGIM4wTng3hD+61ro9yz5Cxe3zlrgjqddKrbWWLE49MTbvYHLGpJzKWIgtwHmFFCOB2L/5OCjI2l9w5cErwP2O8mgAvep/dSnKFINgfWcu0E8TqXJH0LcopIccE4BEwh4Oawx+3NAcenJ8QYBZicOE5aWS9HDvANhKAirSdxy4hJ+T0DbLu5bSr1jTbazhrrSJoVQvJn9XX4AIkGKHo8AJYsD94FFOckU4QtON0JzEvnQqmOfnNdW/3YqJjOXKsETzC/vc3chzFl1QMVgC1AuVZRklhA1sIoDM1ZhgLHovMqWqyCi7xisKXjVrbbiXruglVdB9/PrV8Zjp9AJ1VTRfQdof4T/d0Ylr4ER21ZN4sHr1F41ftGgS8aNW4DBq6f3/fJuFv7qci5uS4kB1NSkadaMjvqzoBEzWtKE4L6jKD6aznnMGgIiOikbIctiBr2IotT9U5Fd/LSauahigbC7JxWHhEfSUk7EoJvei/zDgckcyNYnC+9B6eIlMT3CkQ1dYlch3E6HrGfdjgcDng6PsEVj8KMwUu5dssDb17IRA6FchXzg7LFGCPGQcrCD8OAeVl04THO5zOcc7i9v4NzhHmecXw64nw+Iy5RCknAYbfb4eH+Hh/ef5DMDLpZ+eAw+ACvQcYFBUlzxdvm2NwwNnonCGtwOvHMwif+Rp1EQJ1gXEFFGUXRTBEQ4CMIMyxDBhfJ/97Gv3TsB6gzf6NAbhsx2Z6yOm9lfa57P7f76Nwnmyd6tBgV1CEXEqtYIM6dztaWitKWPtpiAZ0jKTjiGV6dSesaM8V30b9dt76uLNWL5dUd1NwzLi3xz7WXg9Tqph1gEaOWIlr3DKYUByDOhdIzgHt9ybrz27utB2FtHdx81fpURRzq+tkOLWsrq3zGwgycxmqt+aAuAJKJCk2yzyrOmMJachq1GEanoMo5IUZhDuJ2ECU1MIuokZJ4haNIfiP5TJ0ZVXnJJB7PJq8UgoilqvuyuF8iYBhUd+MEcOe4IOUEYgGWxhaBGCPu3zyACZjnGYrYQCEwa/pf8pJuhMUDm5DhNUXkbppwejqKWJUt7sxhGDxCcMh5QeER0xCq0eD49IQ0L1jmGWMYMO32uD0c8Ob+Hj/cHhCXM1JMAiZcEBxhCAOIgFQInDUgWjeCYbC6fZJaxzsCW74m1uwKbGxHioMG9Xw3luKIRZHfzceUMuI8I8YEOMC7ARSCvm9hYL2/HvfzXCcdddIFFPDNHaWAJayutO/sc8kNDZjBZQui11hI0QIQFST1emqPkPOIgZLh1HHIAxIHrqmNQUHmWp90UsXNDPE2d4qKlhOrZ1RbQFrrouRYMyDwtsLqM+3VivPn/t6ymmsY+Wnc7EQ3tgcW5fv1+12/Ytu0Lo+vyn+9x7bH5LowgnohMzU3fyyz25minJRB2q7c+mI+MaiAVDQ9inmT55zE5K9iUUpJxMKUQMwVOEGklXftOQhEovsqMAWt9jnIzjjqOX4YsMQFRcW5YAs+ZSzLgqenJ9zf3+PPf/6zKFtJF3kRlwAy6xGMjXBdbOMwIE+jODqam4Wu0f004XDYYTcGTJNYDHNe1LoZEWNCTBETdtjv9ni4v8fD7R1OxxMe06MZyDQ5noyBy0WziEo95eADdrtJCj2QOLs6Vr0dgCWJWwcXrsnwHBH20w5BHWw9oYaU2CISB1ixWKYUAUfwozhBkpUgRoFV6utF9ooRNcuF6/wMYfRUxhCkieq6eQOI+4llKu1m6nVXGmOE4kflq+jZVDIW2AtlkSuwU4W7zaGSxdAikQOauI9NN9VFk1RjJYt4uHoC6+N6/Yr/n937Zb5Sr9dJ1U5QJRm8+rz9Xv9XB2N7zWsgwwAymvcycCn+8TO/b49sf10F2WsvWs+rqWS6+3gizVUtql/TFXpTVrJGrwM1hQZn1aDkTveUlVGWIhVS1PxdMlAyIyUJbAYKfBkQ4DR7wlAXBMNDAoIt5YtYeaAKc9bF5IaAUS1zOWcMg2RkAMQitywLHh8fMQwDbu/vMS8zmAAfBlVflNUQm54JyhiDI+zHEWlZsBTAE+MUzyjZ42Y/4Zs392IVLBH7MYA4YPQMzgu4RHBOcJwxEuPusMP93QHvfwg4Qyazcx67ccThsAMXxnmJSJy1qAGJ/g/iMDp4KROWCgM+IDNjSaZv4yq+MoBpHLCfBsRlgWNZlFkVQUyqjC6ivyGLZyO1opG4g0hMoBpFOCMXSdMs0qikrTFrYT9b1/wcIokoW2ZmuKw5s7KERzHWi3k7l43hy5rRQloETTNjxYBssRKg3goMknhQY3JOGbpjKWvmRCer5hNVOxepWGQSBEwL046R2hZN9O4l4FIE/LcU4VPtFSC1vegVRO+praI2q8G8KqSqYmqLov31NWkYWjR2e8zu+GeecwtpvSXyuVaNAFaGaANwtW6ZgjJTy8xqwq+o2wVZC4kHu+kTMrRQZxaLFEMqIxv1LcxILPqWrMruwhlDToAT5fkwhArcqs6GDwEUHEpgcDKQ0msA0HwwAqTOwQ8BfggIISAMA56OTzidz/j48SO+/fV31cN9GAYMblBQEv8hSxOcYtYsmAUESR3iA8Fl9XouCc4Rbg4HcI6IacE8PyFnUcoPg4iB4tQ6I6cFhIJx9Li5mbDfD3iaBszzgmFw2O0G3N3sJXOmIxSI5ZBcU/I7KgKs5EGZweRkA9CxLZpDnFhyLg3BYT8O8JBahTlGcBBQcUSgIBkZgg+6+2eIw6k8qxRcNrkHqptKYCQFQkJKzU9M5hEqa6nbYWXFmk2CmjWQOW8ASkF0s7kaOLAW8QC1DKBAC7utx2v/bIU6cuKSAGgQu8wZBgkrrWAkag4oG62l13oFibE0vY88r6wLsXZm5BKfXYfX2heBlDEks3ZZa+ky5PGtnNNKhub1tS4a2bW64yqvdNhC0Muk2s0tOtDqxTrWGzeM6lMYu6ow7m9OjtTCA3srVeLLpkhFv7NwBbrCpdbIK9yOsaDSwsASE2KSElEujHWXgiZkYxB24w7eeRVNUg36DSFomE1BGYwNqMe2xhY6L+XIY0z4+PEj7u/vscyiwwqDxxAGxLLoTi3VaGYUpLLAQSonO/V0DsFhHAICOSAnjN5hvx+RHxecTyek4LDbHZAWh+CB82lBiQFxWRDTAu8YuzHg4f4GcZ7xSMA4Boyjx243AiCQdwiDx5JyDdtZYhY/dwewxrfFUpA5SYFNrwHTTFW8c1wweAKNAafTCSkLnfDeYRwGfc8JKRNcQnNdoBZhIZVsUJd7LRulzEuYsxMdkPeyr2ppq6JzznlNtscFlmnJxEidBZtZ3q0dVhGNqAKcI6Bks8Dn1dy72KxVN1xcAVjSEBd9JnFdgFj/zNXGaf9MPJT4GzHIqNpDALYPjyurNZ9r4DxXh+HPtVeA1CYuDzag7VO3skA0gNIw/nam7iirpp8JQFC9prEo3UMg4RqXp35pq5YhKw+ENiHte3MFWJ2nFg55OX2WUcDMwqwMyomaXXmlnqtl2iszM6IJsTDCATk7pJzxdDxif3OLaZoQplGHUTyucxZxYwgDxnFESgnn0xmRovRNK6MYS3TqjOmcgwsB+4OY9ed5ltAPADe3N3h8/Ci5zkNA0ZCXEDwcHZCXM06niLTMOB6fMHiP/bTDMHiMo3iOp0W+D97De0IuCcgepSTkIj+5iMEgpQXzfIZzjGHwuL+7BbgxkBAchtCsrsPgkQtEh+g85iXi8eMjlswoBGTvwFnq/6WSq1+X8AdNtVuyZquQd1Y4axoUAUaCV78oreBT52PHfmDhWqID9J7BLGmWkzJgEKuriYYZVR0lBKQyw/leXwWknMTQQermoXJFMeuazRczonRriVVZ3+soDRT6OVyKsEGAJWGgY5tSCkxOWJUr1XdMysMLm3WMJlZ2EkXDP+tnaX53JL5xOSXkkhHjyxjVK6rF1NwSQm+5/VmPqc6NSiZZ4eVCpfTCz9AeVliPZST4lAds7UJr3cBdF/2aBQ0wkDKFeBdbqM/eMyqnaXKp3kgnhgKzQyviaUpzqFVo3V1lniT+SB4eLqjvDosp/+ZwgHMewzAAcFVZvSyLlDX3QZP1AZjleQfnRDGvotI4TuJuoQneGIy9ZkJIKcE5h8P+gPP5hFwyBh8QfID55QTvcdgfwCXhXBKcZ8R4hnfQeydwEevlPB8xTiP244C0l5TJQwiwoqCOCHGZkaLkYvee4Bxwd3sjCn9yiCkJWLEo70PwoCQOpOO0g/Me57Mo4/NZnF29Mp2cdUFokQfAwWt4CCtIkZeiDIMX1hu8+BmVklFSRIwzco7imoLmVlP1VGhinHcOfvDIvsBnqjUnc841WWBzlzAxT6yUfRK2Ugo4Fw1Xkjkl7h9cgVOP1GVTdQBgsrnVBc/r3Ewpi46pS9XC3tQcRXXrjW0TNZYE0vhdpylpiGv8JFhNJp1/oT1H3gBuSknCvwo6cfHT7RVl1nVg+vzRFTbln5oe1nViH3OXFxkVdq85uH36/s2ceV2Zftk+d8QasGgtatbYplKrejhqLgGA5uAh0rhB0n6K5zcXIHNGKQmEUidqWpZK7Qt3SmndXcVj3dKVFDBa6EVMCcsc4X3AOAyAB7KmFY4xgsEIXtKrALIjM8tnNXFeKaLQHXwdT6/J+cwCmEvG/f09np4eAUAq2lgpI2IM44ApTShlwS7ucS5HAQNPKCWi5BkpL0iJsNsNGIeA2/2h5j/nxBrSUXCOEeM44Xw6YRwHlCTxhUMY1PdqhCMS/RQzQhjB3uOw3+NwcyvK8TmKS4HzkKrWIqpJYHCWhc0qhmgg8zgEeEdIOcN7h53fgQkI3qPkhGWJOJ2OUmiUWfN4eQxBQJvAgFZmgRMxx4Kfgx+AccK8RJxOZ8zzUllDCAHeD7rxOVjmg74ak+kVCcAwBs2eqnF3Wdi4SCptnouYWS8AQFhdU2mYBTjDMC6nAh8kQsKx5q6iokxfdVDi+wKz3omjqRgTuIt1JIiflVP2CMgmEZcoYVHOwQeHeT4hhAFv3rzBzc3NZ1aotFeAVBuAalnk7ktTktlqgyzYrUwKrAHq2t+v6hC3K19YPezzZy/BDf3JRDsFKTJ/li76nprVjwjq2Uu1oi7D0nEoDc8FXBJYE9qJp/mi5lx7Bmqg31F12z29t13NYVkiwhhBZwewFBkNXtPBeKncEjkpCAURMQvDBcLkJtlEOt2KBOqiBtYOw4Dz+YSUEnaTVLUpKdXxMzcKB8I4juA8IY4j0ixK8LqR6cLIJSGXpOlkSBO4MmKKOJ9nLEsUnVvOOJ1nVdonKZEFlvqBLJP9PJ/hvMc0eTgfMIwjnPdYzmfEFEWXqH5TThe/gFSshVQtg6xzGpvoCHGOIOcwDSMy1HGzsP7MYq7XdzBqWbIQHErOiBwtYgWAiunksJtGDMMI0Anv37/H4+MjUsqihB8ZISgAaKGKqgslC/shHWthvF7fP5cImD4YQDNCyfwVxqMSgB1T51hPDpTpEKNoqTLvvSjr0DMocfx1pdQitx4ejUA1FRABABNKscrmEu2wLPKuiQjDGLDf7/HtN9/iV99+h4eH+2dW5rq9ooJxt+XrDyJFXDZ9i8r9BcqeCjp0W4HRNSb1EkZlvhpbZ9ALKKLLj55vDHPtbaKcuRWoPxQai+r/a1kPSXUOrlJcSbUbUeKCXLJ4ms8RQBHRgMw3RWMDa15q0y2QpkgZNJ0GaZkoRkoJPggDssUBpeVFWad5qBdAF5mwJ2N1GeKS4LyIjlJdecC8SFDwfpxw1LxXYM1nlSMcN9O7Y3Uc9U4zmUiOq2EQF4LHx0cBHWZM+z08eczzjKePjyA/Yl6i6t1OSNMA56SsVs6MlDNSFF+uXFhyrKcCHwIYhMfjEx6fnnCaT5LsTTdxI7aswdZQ9wMHdUYkknTFyoSGISDsBmBZJKNCGODdgDiP1Vw+BI/9bsJukmeReMRYx3oIAeJIyhiGAbvdhOP5iBhnzPO5WnF99pJSxXnACcOLUcbX8pCZvrM4AoLmgjLgIc1u4aCuAOj2f1YSY2s0a6rpqoBBtSsSNM2wa/nLMlBK0PdIzW9Qf3ovYU0tP5rFzSroOZECAKlZuSwRx+MTzucZd3e3ePPwFr/97W/xzTfvcLPfYVTG/7n2Cib1DNtZYVeP3iobG+t67rwXA8nXaWux8fLmvUNms+Rc/ie7TPtbz5bwAvO30RLlOUYk9TrPMcpLdR7YVGtxzkrKC20ullubSDMLKH2HRu2nDA7X3ovc35SubH5YziHoQihZwntSThoMLf11XnReKUYJOA4DTsuMqMn4gAJSj/lSUntujVvcTYzD4YSSGU/HR2UxpLoSYD8dZKGwZO9M6YjTUVwgTierdEJwPoAL1fCZYZRip/MyIxSp5XdeZpzPZ5SSNZg2gi1ek6GB3baxqSMuSTI/KWpKmHYThnGHYbcD54JxFAtkzgXLHDHHBQCwmybsdzt47ytwxkWYjeksQxDRbBpHeOcl5c/KgsWa/4r1fVssZEGSwENxRu18BkSPI8aI+izmi2evvgZFsxbaUBUVWx4tRq1n2Sa6hNAQifWTpUiG6CpNb2XsTudjFhYlbhoOvYsEkZ0j4x+XBcfTCfN8xDSN+PWvf4V/+qf/hDcP7+AdkPOC43F5bpmu2itAyiZk7xuBWmCxCSySggMMJd3Xm7jUP6fItuubXaxd/0vbVQ9drJmTzuK6FRsd71nTtty1gJqrL0dcDIzGiXecpe+VmL2oHt1iFnZE4Gw9gcTIed1piy5wFQMNTEzhyNAI/SSWGrmviJvOK2NwXlmtWplUtCMNLLXYNBMSABHnvCPkHBGUfZ3PR5QkCeu4JCzzGTlFCeWJGRS8huM4jMMOzB81TMZSuziczwuCGxBcqEzRh4BYMuZlQYwSplO44PZwj/1+jxgTlhhB5BEX8QVg1uM0DCeEAUuMOJ9P8MMeSyYpYqFWScvtbXoRAktgsRtwmPYYpz1cCOCcMYSAcRhxPJ5AXDB6h2EacXd7wDgE1UNCrYOmp5U574hFTB4CTueTisGSf1zAR1hY4YLBO90MZF2lnCTAfOKaJqf3CmdoRIRzq89QV0he6anMulctgluywBC9KYpYSinBk9PkhQI4XvVwUKYsDqYAsoSExRiRisQ1Oie6TeccoFWFSkp49/Ydfvfb3+L3v/89vvvu1+BCeHz6iPn0CKKv7ILQFnnnJlBdDtp3V3/r8KH3TdoCFGELVFAZW5mJ+XsAF9dc91Gu9hywPcei+jOtWSCm3Ed2I1vkFaQspILFwbMoG2NIsrlFg4qj5jB3RChemIwnTZcB1FSutnM5N+hCFvN78KHutMaqSilIvKBocjjoYnRFqLf3oYpbbGlrVxZTMZeT6h0AYV0xRpgS33uP0+mE+XTGMHgQZ8yz5DgPzmOJM3KSYzOKKHg7XYv1qxTgdF4wjbtaa8850W/lVJDVWpWjZGQI04hpv5NcVs4jlYzdeJAS9Tc3ABin+YzT6dwFaDOWecHHjx8riKkbbi11H4JHyhkxAvv9HtM0wA8jvHcI3oOz+PMQEfa7HW7ubnG4uRGxWsWycfRIrs09KwJ7OOxAVDAvJxQW0T4M4jhL5DRJX4L3hHEMlZEumvedHMN50tAlruNvGTVCkPebckFWcdZClTRzzYbd65zvlbf6Sf1X2Z3RAeeUUYdQfcIKSy4qZpnTOUcsy4KUM+BkzoTgsd9L0ZHD4YD7u3v8wz/+A37961/j9vYGzBlPx494evwBMS5VTPxce4ULgjTitnuAzI6wbqamu+b6/rySXJlH/ZO6zwFLXtLdenXNLeYUtbKBurgjXAJaUyRaNJ7eu8tc2J9XWUiXzpa7WmaWH1x9c8XfnsWxMxexNEniEgGiME5aRUYc8MIwYFQgEt1HrhY40bNwK0cF9UomJ17h6i3PWWMDM4EGgDS/FZND5tyevUiKEgexRHLMIGVncUniYFoKxjBgCCPen99jPhfxg0oR59Oppjj23mEYB03FEhHGCQd2WOKCFJNahLSM+zKjOI/xcAMQYzdOUtSBGWEY4YcBTMA0TfDe43g81mIRh8MewzggjKL7ijnVdxTCgMLAPC9YlljDg8w1psYfliL5uzypQnfAMIzYaQaHp/MJKSbc7HcYdhMO+z3CONTCq4Nj7PY72XB0M5ymHe7vbrEbpZ9BvfyDcwiHvbC9ZUFKEd4PALGM4flcRVYLm5IiHoMUwhg8nCd4T/Dw8Ow1zY9UESqlpfXpWVQDqUuFucmKAmpi8ZSsG3KyI2BUUaiwOnNyAaseL8aIuCyYl1kzbMh6m4YRbx/e4Le/+S1+9d2v8N23v8Ld3Q28d3h6esQyHzGfT1JjsiSk8tWdOYFLJN7wxx8lmDVNt+klVleusUlmGdt0bcuO2Mgw9QaQz7Ko2puqX6Sqc6j6kl5X5ZrPMaHnjyaSqQDMEF2CF1P5pNWMvfogSQ4lUSaM6g0+DmMNcTF6752vfepHzvLG2MRtDoAi3gUvu6I59gnAmT+b5FU6z0fkKKmJUxZP95ILTiC8efMW77//AU/HDziXCKepkOfzjHmetf6f7PZLSph2exzu7pCXiKcnUZ7GmBAzI5WEhBPeaH9CkFQtYICcR1Z/ODjCOE0oEJAOIeD29hbOezBDXSayOqJmeOcRY8bpdBJjhDjjyCLXIN8lRngULDHg9jBhnEJ9FzllfHx8xHk+Yhg89ocDhkmr5XgVnUtRcZpQgjChEAJubm5wc3PAOEmmimkKGDWw2vkRIEJKomR2DliWM+b5jA8fPtTsE94ThkHmxm43aeiSFHF1XnJrySQuOM0nLHER8Vetf80HS2ZF07/qvtutlToHID5ZkkAwg4vqqWjQfGHC+ItumnM8y3jHiHmRyARjyw939/jdb3+Lf/rnf8KbN2+wnyYwJ/zww5/w/ofvwazZPlT8L+mrBxg/07iJaVYTr/9u27ZMpjEUW2xtkM2ZUpSerta1wyYy/HqTF1Z3kOKUWDWhsn9Z1Q+IFXI0AyN3Snar/gpSa5zmH/9sP4gQhkFyJ02Sj2hWkSSlLKA0jhi86ohiRE5ZPMyDl50+tGBVySDaWGVP5VlZG7M8Q1xEOSnZQp2CdmOODgxmcXg8P0mpKgLU01zEr5Pu9N98+w5P//WD+v4ccXdzoyIMkGPE+x8+oFbSDYTDOOH2do9hmHA6nvB0POF4PoOLlmv3HiCxXt4/3KuzIeN0PkNcLpKKOkEtZrsq5j0dj7Kbz7NY/3JCShnnk4jUBM0RZspy1beUnOA4IEXxw7rZH7CfZLM45wQHseQF7zEGMTCkUjB6j8GLd35KUsiBhgHMGbtxxE7FRouvHII4f05DgAtBNqHCEoDOjDjPOM8LzvMZOWV9ThEZ9/tRfMoAtbyJ1dAcixlc9ZvkHByHJuJ1EkJveS5onuky74WBmhEmpQRVjlb/P+91tngCikPhgmU+a/oazWKRM/Y3e9zf3+P3v/+P+Mf/8Pf45t0bcCk4Pn3A09NHfP+XPyHFBcMw6EbvNO3PZ5aOtlfrpK7pjSpb+Uxb7f4r8aspritEqR6j/S45bowV4QpVXAFfFb8at7Hcy1V0pQ7EoCClOXwEqAB1grIHRbX+0UpAXI1F7blmOZx2exBGHB8/Yl5mLOcZs/oFOSIktvAaFfOcX/XPFp2JmdMwAgSkKMnh1nqmpqvKXXI9rw6SpCKjc+K4mIt4AOeYcD6dMJ8lM4L3oSpd53nG6XzEP/79P+Dm5hbH0xP+8v0PIAbGYawbVYxJ9XcF87wghAH7aYdxnITZwCFmsYxxAYZhxDhNGMcRD2/eIOWCqL5TOTPO5xmHwwHjKMxif9jDe6cLJGKJ4igIaJiPB3I+iTVUF4QsQilCwaoUzqUg5YzDzQ3ubm8xhIAYRcm93+9QSsYyLzidjigA/CDpYIYh2Has1kwBonEYMU7yn3jnQ/U5Yq0jcliUCYtVTJ4zLiIKZ60o7fwgC9mZn5QYQcRgIu8/xSS+bLXcvbDgohu5FbIAqKXBNvCq872oF75sbjHJeDpYmXlC1rAsr1JLZqjbxRmLbgwFMre/++43+P3v/wP+83/+Z7x9e4/j8YM4ApdSHWLHUaoRkc7RktbZNT7VXqWTMoW1MRJblE3QMyQ3WrRtDdlXn9H6ZxNnrl1EnSXJdQC07mX9uJ6+vlYNf0GvCes4EUMTjRVQLdNFeiJdXK9/FrPCmc9L8AGn+Yh5fsJ8PqOos6CBS2b14xEHKACSI6k6wtYQVs1NlRlxXuC8BBAP3sMFJzmoYoJUNxHfmlIKYpJio0SEZO4FeUFhFrCDWB5zFB1TXCQZXSnnmskyKrV/c/+At+/e4N/+9AeknPHh4yPubm81pIN0si+qFPUYpx0O+z0ACfI1v67T6SwVhUsWnVVheB9wOEyw8KDHx0fJOe4Jh8MeIXjNSCDJ6MCoTGG3mzCOA56OM5g/imuBDzinDHIBpSzqhsGYJvErSznh8ekJOYlV63Q8wfmAEDzGccB5PkthCSLs3UEU/IOUO5vGG+z3BxAYJScJpfEB4zhgCANSEj+4+7t7fOBHxJiV2QDsSN0CJEQqpajbZoB5c4vSPtQ5ZS3FiHmeBdxyQskFecnIWUA6hADng849pyoSvQrBotiFHaakYVWsXuFREhoSYRoGBO/AOSMTUJJGS6Qkfn9FfOoe7t/g17/9O/z+n/8Jv/vdb/Dw5hY5Lfj++z8jLjOmcUDwgNuNCF4y1orzMyFxAg3jlTV02V4BUk3HY/8DqHPFtwGletRarW5sCZWyygU7RlVBqpn5uVGnjZ6qKdkbG1qDH9l1u+5VcbPqtYQKO+rZojKTTvlF5izp9KfF+3VAyR2FlcVWtOZbS3finJQgcuSQkbSKbkZMEVwGgAi7cezE0PZYtYiB9+DCUtABrGLCiCEEnM9nJNUjZK0+k5KwBPYejgrm8xlPx6MuKi87ZJzVMzrKmGhlmpwLnh6P+PDhA3JM+O/+d/8dfvWrX+H49ITz6YRcWDzTNUe6xR8yFxzPJ+zPk+ZZl2yjkro2gErEOI7wIWCeZ8zzgsPhgN1+wrwsOJ5PKJxlJDVzgHixi1vBOI0128PNreizYhQlPNyAU8xYUkbwEijLAIJzuD8cwJzFaVQdLadxxDgEpJKwpAUuSBjIEiW9sSNg8B45iaL8/uEBh9sbyUgKK+ipecUA5JjgIO8xTjvMy0dNvSN+aTmLN35MUdLxDIMEc3eZQqvOkGQtZY1YiGlByvJOT6dZS50FDMMIpxEK7fxcM7YWiLibcqp6RJtjMYqxwzsBuwRgDB45kVQ3SouUXisZYRhwuL3B/f0dfve7v8Pv/+mf8avvfo2bww4xnvDh/feYT0fspgGeGPDNBQbMCE5BlBw4fE5VIu1VYTErvbT+2wlo3YLeinLdISouOWNOdEVguiIWGkNa90F1SBuweE2z6Ox+17Frdg9a+9JbTcwXxawyJUkyOGMwWZPDbZtzwipylkj7mMR/qSSZ3Gmaqk8TnIOFo2gmOwFUomqNySnhXBoIAkBahFmZ60HOBcSMtBDSsuDjhw9gLri9OaiOaMHT46PkKx/XCfaOxyf86c9/wsfHj/jmV+/wD3//d3h6esRf/vRnjMOI3V5SxTDLdYZxxKL6ovP5rH1gEIlZ+9fffYfTPGOyAOF5xtP5hNt0i5FFH+OdZICQ0B2qJdGdkw3OOYeYomTmHCfM8wzvCbd3B4QInP/yHqNzGIPTFCMFu2nCw80ep9MR+4EwaraEaRjgvccf//xHpJIxjpOKyQkhiMsCEXA+nXB7OODu9hbOO5zUMkrQDCClICMjZzGCjNOIXUzA+4LlfBZFs2amjDEKi3Me0zhhv9tjt5uwmyRP/EqEd81VJOlc4SLuK7vDDuO4034GyUuWjSXlftIhpSQ55o9nEbl1XomRgRVHCkjZs3PCFOMyo5SEaRrx7pt3+NWvfoXf/Po7/O53v8Nvf/s7jNMOaTnj6cMPeHz/F8nXNQWQZnyobhIUEJwHnLnGfGWPcyslVNU32FjJyAGwnEu68C0rArUFziCIkrqPgF7D1HW4cU3ehKa01VPZZFBjU8aWrl59fdvmlLn+fZv7zhzlzJuZVey0BHU551pQgQia3E5vQ5aSRX1RiDAOY3Vj4JyxaGqRGBcs8xkl7zSViaSEzaVIdgTyQPFtN9LxZGYxcUdJODcMI/xAeFzE4ZJTxpyT1PgrBUs843w6oeQF+90OrGxvUcvN4XBAGEc4SGYAoODp6SP+5f/9L/juu+/wzTff4XQ8g4iw2+0w6cKOen8LoTidZgxDwDhOmMYJOTHu7u4w7g8qmgT4QRgkqSOtlILfa0iNg1cFtgReO0BdLOx9LfOMkoU1DcFjiQmeCg5TwN39Df54fERAxtu7Aw6ThysO37y5xbdvbjEQ4DT7poSuMJb5hNPxA4iBQUHQAZhCwO1hD0+M48cP4LiIIpyoKsRRpJDF7BYEghY8LViWWWoKQlIF5iIuIzfTHjc3B+z2e+ymEfv9DtMQkHLBUgxEJCi4+rkVFSffPeCbb76Bcw5PxxNiSpiXiLjMrXgEzAnYYZlnnI5HrW1oKZIjAKcOxro5e8Z5LsjFgTW64LDf4bvvfoW/+4e/x29+81t89+07vHv7DlPwmJ8+4P33f8KHD3/Gcj5iCgOSinbFCpeAUVxATjO4SKD4Yf+VA4xbziTTGakIpPoSY01WOYUASw8Oq1NWz++V5VfaBUjxpceVnc31msLruL4aE/iaIGiWvc9a5KjPIdWJmCp2yO/N6VGCiiFR8cUyY6Zasmp7bafBm16LPlg8maQ4WSRbpU4OmfwSGJtTBpOIcRIq00IoBFi5hhsQiVL75maHJTj88Y9/xPn4JCW3Rtmp53lWcTCBNM5O8jpJJeD7hwcwM3a7CdMkbOW//eFf8f/8H/57/Jf/8l/g/xDw8cNH7Hc77HY7vHnzACLCPJ81Lkvi74ZhwDAOYjEsEv4xjEEmMNTHSp9lHAekFLE/HCQ7wzipgUFcCIrWgctavML0K2BUEbcUcZbc7SY4zvCu4N3DLR7ub0BgvH24xTfv7jEEh1IiluUMCaCRIOnj01GMDSEAyNgfJux2Iwji/CmL/RFEDoMXD/phHKUSMrN6tQNLnLUatVy3ZbWSFzaMAXdvbvFw/4BxEH2WuB+MOJ0XCUXion5U4mdmqWPu7u7wd7/7O9zd3eHDh49IJWP+sGBZxGJoQdUMFeGSuGbUHE4EwLLmsnKurPJRSQAFcAGICsZxwLfffIt/+Md/xN//h3/Ed999h9vdBC4ZH3/4Cz788Bd8+P4viPGEcQhAzjg9fkAp1KJxSHJ9RQ0pmvY7xHT36XWo7RWFGAb9aVl1SCtybEW8Xt+jzWl+6AoeL8sjY63L9aWdQSeSKYCpR678oQp06iCGTH9GNXUPrf7XWUKIVGEuuqeaEZQBiSFAVWSX3KpfkJbrtgT+OYlOKLN6YfdsTYM0D26PkTxicDIB06yZExagJBC88FMlZgVJ0wQTcoi1gACR+AURCCVJnOCJj4jLAQ8PD/jm22/w//sfn/DhUbJvOg3lSSmJSFYKzur17FFwngvokTCNE0rJmMYJYZhwXmb8y7/8f/C/+V//b3F/94A//Lc/YL+b8PbNA6Zpws3Noe5Hzjkcj0cA4m1eWN0CuAhTWGakXETJPo5IOeN4fAIzsN+JI6caOkFqyAAko2YkoKQoMX7KTHIu2O0mnGcJ39nvJ8TljNvbG9zd3qnyd8Hv3vwawQecz0eM44jz+YglZczLGQxgPp9BhcElwhNAJcOTlvpKER8fPyLGBeM4iD4oiKf6vCywYqAxzjg+PuI8R6AIy5vGgFgYGRpyEjzeffst3j68EUNHijjsJhz2B4kNTDNizkhLxO3hRqrVsFSI/vbbd7i/u1En0FJzXJ3OTzientTfqWWTFQPKIk61mi1C0gzbuDYrcWZR7hOkGtBvfv0b/P6ffo+//4//Eb/73d9hNw1Yzkecn97j6f0PeHz/Hmk+yTM4xjyfsSzC0A6HvTjngsBZVBbTsMNhOmA37F60/l8u7nm5oCnPAWj6WlGIWd5mM4C5Bg8yGGAdFEie6Yo667AZoM+4YGJV96U6c1aFfYeIrLoaU8jbN6b/qgChD9LEuw6gutg91oBfc9y0HlnSeUvVKwnVxGFQTMJi9l4WqVZcWKPMGfDQsBVNohbIw08E76QUVUoEz+JYB5bEbL5K2oxcSMGxQFReDPOfASyYOCEmMc0/HR9xPh+xv73BP/zHf8S//Mv/C4+nJ4BEb5AiY44LfD8GED2CsCrJ6jCNA+5uDmJmyAn/03/9/+J3v/ktgvM4Pp1wOp01GR8wjpPmTZI4xBgTlmWBCwVjkKoxS1xAp6fKyG8ON0hxwfv5CTe3tyu95jSNWGbJdeVJACEM4vAahhExPaGUgv1hj3lZME4e9+EWwTssC+uidHj8+KjBs9InLuJhvywzlpxxejoC3uE8n7AsZ4yj+GWZM2yMC8DiPOqcxHWayJ618k7OYrFdloinpyOeTkcwMw77PTIDS2FElnz34zDg3ZsH3NwccDoe4dhjDAOOT0/4/s9/wcfHjyiQKIT9fgdx8wgYwiCJAb1HKRHTOGCJM+bzGY/HJxzPJykKghZm1a8uWVBSj9CRBT1D1TBFy84D4zDh22+/wT//p9/jf/Wf/xO++fZXONwecD4eMR8/4PH99/j4/s9I5zPG4OHI43x8wvF4AkC4vb0TFj2Nqt8EwjBh2h8wjCOG8P9n7b+aZcmyPD/st4XLkEdfffNmZVZVq+qewQyJwYA04ImGL8AnflDwhTQIEtaAwRo9JbpUqptXHh3Cte+9+bC2x7lZqAELhjxlp67KiBPh4b58rf/6ix8Zk0JnHMDieB3Lnf3haxrzpm2KnGQy84uBm5jZH0ZEENLkJ88RkIvuk7+Ynvzw+wnsJkw9HYcu6QFdnzqiT7Z8avJinjpCDkA++gGX4s/8t9MLmJjfwkMaccMo3tbKgHIHvEqKSDRciyeGilwWHXkv03+rtYhNrQarA/geN/S4voUkianDopvywYEbYzFy9MNA33fCTNZTd9Tjo8zEO03nPdvdhqIoePn8GZeXV9zcXMtdMxL5xogfygk94X5KyI8KssSgikzsdrXi9vKS9XLB2dkpu+1O1vUE8X1SGhBy4ny2oB9kDBmGgdRaJGlXXDnLTM6rvChoakc3jAxdi9Epi1lx2ByOwyA3gUS4Np96uaeJxY0DRZ5LZHwcpeq6ElpHEN+kYRT8beKRGW0O55EfJYCC4EVobKwEp2Y5eZahlaJtG9LIqwpOQHMm76ng6Nsu0jZkVpjG8STRKGPxDjIf6L1nCJ48z2UD2HU0+wq8Z2xbbm9vubm7xSQJXilmRUlqRcfp0kyoI84ztG286ct7vLm9pdpXUtiUPpA3JRBWyELKSSclVsc+Zg5Fbym8BEp46XiW8zmfvXzBlz/5CeenJxRljhta+npLt9uwu/1Itd+Sp8LYb1uxk3ZOcMfVcsVyuRBMMQRQhjTNSbIcrcTz7C/5+suLlIkmaurBmTMcikNkGk0aqcOFHV0tdSxQxAMViL+GT+xOP/3603kxfv3Juv9QtOJDpE5NJU86JA0HPGP6Z/2JJ7l8ylNXpX9QzCTe28S/1w/6qnhST0CmvO2H9/EDnVR83Sr+6oPEj6totyFg/GRzYbAmBQ9ayTiUZZkIi5UW47E4XoYJVPWOrm1oG8FQDpshH7eGTIEYgc2m4/7+jsePH5NlKW/evHmwtnVetHLWRAxQxophFNM++fKkiSbLBNS/u7nm5YvnfP31t5FiITbJaGhbwWOKsmQSSU80jL4Xm9/EWLKiILEJPa2Yz6UZARiGnpOTU4L33O3uJCxUqQOm4rz4PInTg5Ywh0EY6i6+3jzP5e5thITpvYuuDvE80Fo+Gy0Y13qxYN820tX1fXy8fHZ9L+t+GzvBMZoBKoAsQ2lhfbthRGlDmqWC6SnFZrNDoZgXBSjDvmkZmpoiy+jalrbt2O62uK6na2q6tiWN4uoRRTkrAairCoUQRbtWmN8qwhpXl5fc3t/RDU7Y6jq6PUw0hvCgb1VMOYrTtDGJt+SctTZhPp/z2Wef8eUXX/Do0QVZYunbiqap2N3fcvPxDe/efEeaWpLVmrqp2O8qlDIcHa0oy5IkEX2pNQZtTQzglVE3zzNs+iPzpEyaCrMVscs9iBrhYcaLHZCO4xbKy6iixHLEKIMPXszHCDLS/ImZiwridCntof+kGn5aAJhOj4dScMCoIuY12Z3GAvOweZTXqqYCxQ9xIqUnvOhBCxXifDh1jj6SMA86wImGcGilBTsR+w4f2cDTxtPhRo/WBqPVw/sIcsporTE6wWgZetuuw2iLTkBp8fp+wBDk19HJatlEx8ksy2RlHYsUse2X8cDz/v17Tk5OePXZZ3z77be40THGCzCLgPd0nN0oavcksXg3ROW9w1hD37d4P3J6ekZV7RndGM3vDF3bUNcDOvp0G63pnYDmTdtg04RU5RitybOM/XaLVoosSanqPW4UR8q2a6n3e0I8NsMg28qiLFktl4QAbmwPGseyKOgi6Dyfz+OCw7Hf7cgzS1mWjN6hjJI0FBfEksZaOueo9xVTms60+HDOs9lsWS7mpHlG27bi/OCFbJrEfD1ttPC7PKR5Tl1XvP/wgapqWK3WzOdLCAKoh0mgHaRLbOuGer/HjwPHR2uW6xVoxRgU87KM29eRJEkF76r2cj15x66pefv+A03X4hE7mRBjsazRGC0NhFEK4qZWwlHkXJqWTZPXVpkXvHzxnL/++c948ugRRZbR1jturj/Sdw31fsf3r7+l2u84OTpm7MUAUCsli4++o0U80Q8uDiEwhoAaXSTyBpLhR/aTyooShkF8u1E4qccQHpjf05TmposX8UyefI4Eq4qjBB4dRMwqd+7IBfJxGPRelg1Tq8FDgfqPcaIUk7n7A/dqSnN5wDh0rKnCN9KfYDFq8ms6jDyxSBFHuPiB+jAlE4fDz/DA9H8hjIhzZTxKykVagCEMxC7IEQ7gPHJMIpY34WAAQz9Q+YoxGUjSBBemsdMftogTZibBi6P4n8fjFGKAAdPSIGIo79+/5+z0jJ/99Ge8fv2a+/v7gy2NNodeE6O1OHPGI6xQtH2HSaWz3lc1i8UR3dCxr/dYa5gv5kCgqmRbKByfQUzzlHRJNrGHrdMUziBj3BDJhgNNIxfE/d0taS4dmZAO48o8drVaacqypOtaltmS05MT7u43hABZltLUFUliWSxOYmcnNyYdCZTOebIsZXd3Tz/0rBZLoWn0A7udgORJklLOZmJl3HdMZN0sywTTcSNVU/Htd98J+zzLuLq5ZV/VjKN8Lrv9nv1+z76q0dpE7lQXaR+yyS1mM46Oj1gsFtg0wQUoCsGspuNTVxVt25ImGX3f8e7ykv1eRltjLEZJoIXVijyxB3NCT4ApPcc/iNanU9Aay2xW8vzZc/7VP/yCzz57yWq1pK0rrj5ecnX9ARVGurZht9sdbIaGcRRKjVF0bSv1IstF8jWObLbyWTgQCxiTYIw4T/wlX385cJ7ksl1w08gDYmg2ZW39UMAoAHXAKtneKEKcCic/qJEQNEYJyTDE0IKg/MPBe9jXHi7kia/0pwPhoeuZIKlpW8eDQvyhs+KwXZtGQW10xH6mwvRQHP90XTkZlx3WCEowiKCIBE5JTAlhkAJFiIC+QSUqep//EIs7lNU4dvogoK5SRpjJkS2sbXIYdcWfWroTE8XJxIuHCTeMhYoYiikdo2xtbm9v6dqWn375Jd9//z23d3f0w0CqhC5gtej9tFLsq3385LRYIY+jeIQPg3B8ioKrmw9icTJbMJ/NRLTcNPTDQFEWzMoZ88Wc25sbccbUIudp25a8yAnBsdvdYYyhzA1913JzfROV89IZZml2sHDxPt5GMs2+qaVb6zoWixUnJ0fc3d8zjuJSYLSi6zuWyYo8SynzPG7cMvpBPNLTLI00gJz9bhcj6KWoPnv2HKVgE10LNELNgMB2u0MZy4ePl9zd3VGUM+63O6q6EUqCsQzOsb26pG5ip5Vm0DQSA+8DeSq+6CfHRzx69Ig8yyTgIU4lw2DR1jC0DbsITueZo6r3dL2EqyYaEhsoknCYVpSH3CaMwdN1DUMvoLaKuKiewGWtKAoByj//7DMeXchrGIeBzf2G/W6Ld462qfjw4R1t27OcLzA2ideadNDzNCNNEvKikM+jFxNDsSCS4qQAHRy4HzvSyuQoC8Z7lBsjpyImTPiIezBdFLErCBLp5FXAqzjyHObfeDfzSMcRJjxoBFwcoaLRHcSL8qGNPfBEJ8D84Wr/5KI/kDR4kNpMbghmAibAaNSkK5o6wFgJVdx6TD9vAqynwqwOqH2sqGEUjtPYEXwv703HHkQZDLKKncifKkgun1Hxh6nIXjkwnhMUE7VhIMSCm0x2wlqh4rGVouRjBJPFWnESEBZxOGyjCNLtEjz393d47/j81ecYa7i9vQFEDqOQVj3Lc9quEzsVpWhi3FXX9fiZdJWnp2d89/03bHdb9rudjCyrFfuqoqqrg9XKMAwkWSrcMh/QCA1CKylYkoQDbnTc393Sdx1FLip72fRl0ZI3HFJUAk5Y4SDunOsjGDzVbkeSWLrBsdncc3J6wtnJMcvlUuQy0Q9qv9+xXB0xjI7VakU3jFRVQ1EUaK0pioKyLKjrms1GCt+syMmynKppuN9uUcry4cMloEmSDM9A297TdJ3YEXsR8u73VdT9QR91ksvlkrLIUcDZ2SnL5RytjWBswdO0LV1dS+fVtuzrmt12z1ZX9IPkGzrXk+A4WcxYL+e43tF0XTTP03T7irET7aiO9r9KyRZXK8WsnHN2dsZPf/pTXn32kvVqBQE29/dU+y3WaNw48PbtW25vrlnNlhyfnHJ8tI7QjciO0jTFaoMbBvquI0lTZqWQVW2SQlDiDe/cnzUJ+D9UpHQ6wyiwwWMiATDE6KZRPzg+ei+x1bLfARXXnCF2UiGyoyUaawpuVCjl5EJSccOgJsxJ5teH6OmHjuawoZuKHoIP6fj3k1/41D1MRWkCxSf8aeqg9LTlA6YEbc8P5S8uEggPYLj6BCiPeJT3Pc51eD8cWm+tJBLcBEWQyiyfkZOO6aG4Tmrzidg5SoGdulXk5EqShKIoJbBTGzl+6iEZ1kQXT4l0ejCGAynA4zgJgTW73ZZvvv2a58+fk+cp9/cb4bqMHovFe4cymtlyIb5UeyM6t66njduoz798xcWjC+qm5vrmhvXRGh8C/SCAdtd1WGvZs2dWzg74UprmwimLm8rlckHfd+x3txA9xObzOWdnJ3JBDgMoHXGk7HDTyTKx4i2KnH5o2WzusFZA/qapKYqCV5+9YrlaspiX+HGg8SKqLQqRIYkFTcf19S15mlFGQul6fQQo9nvpYLJUNm1t29K0DYHA3f09d5t7Tk7PWayWdDe37PZ7un6gLOfMZjN2+z0ueOqmFqKmE3eAWZmRpQllWaC1fP7KCpjfdy27/U48ubqezX7HZrthXzcH8FvjsTqwLAp+9tkzsjTl/n5PWhs88jrHviW4/sCnipnHaA3L5YIXL17y7MVLXrx4ydnZOdYaqv2e/W4jHeVuy/t372nqhpOTM07XRxwdHYletKnwzpHNZ/L6/YgxmqLISTKxsdFGziNJTJIo+k9NJX+UIoWOgCpTjx3JnLEAOWLKB+Nhy2djcZCUMumoFAEXeVMqKLx2wmzVcldVwUo8ktJx9IuG75PFaYh6QcWD4+ZhGnvw2wEOv04bPCYsYlKJH7Z6UpyC/gS7YpK4TAdSRT/3aej5U3uaCVeTDmUqnIRAGAHzA20yRiuhQk2PDnLyPBR7yYyTrlId8D6PkCH7foiYS/ZglD/lqTkft2DhcEzkMMh7tVbM/cXxU5HnBX3f8dvf/pYvfvIF8/mK66tLdvsdznnquqHvO+bzGWmZstRLrNYHa5fN5o6bmyUvP3vJx48fub/fUlUNhEDfjSQ2YTZbRDBfxn9jDYmRbWSapkBgt1XMylnULQo/KE8zVqvVYYRomkYuulGcGSSJ2ZCmovcLQYD5cZQLRWmhCKzmK9bLNXmRA8L9ShNLasXI7urmnuvrK+62O+qmYfFojveeWbQNvr29par2DOMQpTdDdB5IGB2s1gnF1S2z+ZJAtPeNBNNyvkBbw76uZbPpA21doTQUxQnOD6SZeKiPfR81fZpqt6XrJKSi7nrutluub++oqiZOaAKK+2GkKBK+/PwlTy9OqbY7hjzBh8kyR2K23ChGhkpHE0BjMEnCyckZP//ZX/HqJ18wm8/J01R8o4aefhjouoYPHz6w3zccH59ycX5BkWaCI0ZooSgyrNKSK6l0PLbR2cF7nO8YRidGiqMTofZs/ReVnv+dFASxpJ0imVFKYq0RctoQFKMHr328iKU7SbSKanGZlb3y0VJXhJBKK3SMHdcCsMgICXglUhCviPKTaasViw7qEFOtiR1ciKLSCGgbuUIPHZUyOgYZ6gPorI15SGRlKggxpMtPndKfOzCHMhNn7RBHqwc8KyBbMoWXdSxxRNNhstHkgXp6AOIOBQuvDs/n8ZHV3McOaYG1CTDpCKVrks3Uw6pda8OnCSBT4OfoBlAi+/De8f2b7/nii5/w6Mlj+jcDfT8wOMeu2mMSS1bmBGCxXFKWM/ZVTZJarq8v+fz4JyzXKzabHfu6Yj6bCekyzViv1+z2Owgy9gCU6xnWWPq+Zz5fiFtCNOo+OzsT5n7fx9y9BvrJaTRu7PZ7kiShnM/lxpMZ9lUTN409fS9g9TCO0tl5j9YCJM/KkizR1PuKqtpzf3/P3d09m/0e4tiyXC5J05Sqquj7jrqpMUbTjwNZnhOUFjFy0LRVQ9CGbhhxQR8E0VmW0fc9u0qY9IvFgv1uh3eymhejPENZZFirBRLRmpu7O26ur0V83TRs9xXX9/fs9rLmT4yJFJRAohWPzs44Oz3Bx+K7mpdiIjiOqDTBLGdoHLuqkXw9JZH2i9WCx48vePT4gtV6EYXiwr5vmj1Nu6eqdvR9x3K54NmzJ6yWa8ZoAexGT5oksjjodwdHDnH26FFes2/2ceWkDpSQ1dEJNp1skP63v/5yWUxiYrqEEtq8VoyAU1oKlA70SjMSbVvjbktpg4nbPJTH4ETfF2kLKkj4YFAqMtijrQQGp6a1v3/YrE06uWn7ph64IOhJ1hIpB7F7CkQVvlGxWJmDt82hs1LmwCMhFliJmXTyGsJDlyOTnT90d8rrCP5HzzyCePN494CFBWAUX+qgiP8ePcanhi92nIfnjXUrfDLmTtmGIUBdNwzDSJ7nB5C8aWQMsDEHLkksU0TWpx2VsYJL1HUjFIeZIc8Lum7gd7/9Pa9+8hOePnnOze0tXS9FyiuFCzCOjsEFiVsPkJcl2/2Ojx+vOD09Z3u/O+j18jwXm2Rrmc/mMUtvOPCxAK6uriiKgi6q7UMIzMqSQGBzO1BVO5SB3X4nBMg8JwRP29ZAiRtGpmy9IXouee/i2Ljj9OSM46M1zg1kqUUpQ6oV49jiCYzOk6U5WVbArqIsZpydnaG1YGC3d3fUTS3j3PExKMW+qijKMnbjmt/+7vdUTYcyKat1StX0jE4sT6y1uBDIy5yua0GJQ2eWpqxXC4o8j1YugbIsqeua+/sN95sd+33Fvmu53W7Ztw3j4Em1xgSFsXJ2rVcr1ss5Q9uQZClZYgjOYYNjXmT4ALv9gMaTWLlBG5swW8x5/OiCF8+esl4uSBCKTNM0bO5uZNRrakJwzOcznj9/xmw+o6n29F2Dc6PQUMYR/Cgi7CyTcz2I4L1pG5p+iHFfYuq3Pj7BWktbNz9ukSKxME5Xm46dEIwoRqXp8fRK0UfcSVbEcXtGkEKlxkM3FVB4paN4Vh86Lxfjnp0K+BgpJf5E+nDx/tBtU2gOwlcSNm34jxboKYlFH3Lzpg7j4Egw0Sm0PuBn3ruHpQBxsPMe52NElIsbkiBcFg4WLbGrQgqXDx5c5JhFdfv0gUo7NR2FH6wC5PHqE5a+EkXfOI5stxt2uy1ZJmGLgkkNODceTgqhJ0i5J7bfxijK2YxhGA+FX4znEpq64X/5p/+Fv/7bv+Xs7IxhdNRNhYiHO7z3NG1DkeeSJKw18/mC66trvvzpl1x9vGQcR6qqZor6IgRWqxUo2NxvBEQfR8pyHoFwOYZNI+vtcRg5OTlmvpiz3Uw+4BxsgEPw5EVOkRcM/YBzgW1VcbfZMgwdaSJx6eM4MpvPqOqKk+IEBbFjDhNVCOc8Xd+TZxllOWO5WNK2LavVku12S1VX3N3dScHXmmFwLBZLbJLRdC2//8NXXF3fkhczhtHRdhIGq+OywBiD8R5LYF91lHmOCoFHjy44PjoS19RIbOz7gaAcddNwc39P34/c7XZs6poh2qlYPGiZGsoiYzErounhwGCUeLh3DevVnLQouLm75/Jyhx96jAKTJBSzBWfnF3z28iXPnz9nsZgTgqOuG66uLvl4+T46mQqcs14vqZuaDx/eobVmlheR8ydCeB0eOqXgYRzE0G90YkdjrCXNc47WR0yODPpPJpf/2Ndf3knpCHQFJWOKFqr7gGbQhj4E2gBdvJiMkilFpLqBUQUsJm76pJNACxZF7Jcmy16vwCsdcamIwyhzwL+m9fxUeOQr/PB/igcH0fhc8U0cxj6tRW80xadP+jcI0c0hYkW4w2NDxIS89wxdJ12jVB1UGMFL5Piku1OR4PmwoQyfvPapcHw6TkZ9Y3zt0nCGT7qpWBBVOBic9ZEU9+DFHjeU0f1RqSR6XMmFgw+YJGM+zyX4YBzl52iFVob5csGuqvjq66/54ssvOT4+5urqitEN7HcV5azEe+moFBrnA0UpF2ie5pydnvPtt9+w2+3RQBpJlXmeH8TY+2pPlmbRQyknTTKyPKPrWm5v7zhaS5zTydExRot4t4gMcqVg6DryJBGv8N6BN2y3Fbt9Rd81/OzlS4aqid3Kktm8ZLGYH7Lk+q6JfDe4vrmlH1yU7RhCpFh0Xc9mu2G721G3LRerR6AtQz9gkoT77Y53795xt9nifEAbS1VVh47o8eNHciFqODk75vWb7xmGkVU5Y1YWzOcleZ4RvEhkuq7j9n5D8IrL61vutjvadmDfNIxelkkPQnhPlhqOVnNWyxIVxjj+eep2AAPFLEPpwNBXGCVWN1oZivmM+WrFxfkpz548Zj4rGPqGru+4u7/j9euv6fuWxXwh6Tba4B28ef0tWZ7x4sVzluWcpqlpa1DKPbDhkxSUjSOzxpKijCbNC05OTkUZ0Ivh3tj/yBSEQ4STDmjvQIs2aAyKQWl6Ap22NNH7yBLX3CpgUGQonBL3Qq+iChv3oEtDyeaKkeAFkHfxYvbKCCyjJi43D9u1iWwwXcixsIRPIKSDs0EUmkZCQBz5po4qhhRM28cY3jmNpRF1l9cTwmFsCd5HHo9DhREdRkIYCN4d8KkfyH6mDktQdT7l7k/7zMkbW2QfoiwPk3FYTEERANySFznaCAYiQPi07dNYa5iY55OndRLHW3GdTFgsF1RVJaGQ+uH9nV9cUDUtNzc3vHz5Gc+ePefy8gN3bbRyCTA4wdiCl9beWsv79x84PTvj+uaG+/t74fsouLm9kZuXNtzf36OUosjFFjjPc7q+Y7/bk+eacibGcm3TYk4My6V0NEO8YKwWiVKSWDabW0KwVFVHOzjaXnDO4+WKercnn88BmM9nZJEfJV7hfaSb9DRNS5bPaOotJ2cneCBNE66vr9lVe/q+5/j4RGQuKIYxMLjA+/fvGfqBu/sNTdvj/BYfAschcHp6Irl6IXB8esJ2v+Pu/g4/CrN/OZtDgOura7RWjC6w31d8+/1bxsGz2zfUfc8QcbrI8os+5FCWKctFznpVoHHR2UMyDBUiN1NWsMksMywXM4xNMWnBbHnE8viY87MzirJA4anrmn1V8e7dW/b7LevViiQR/aMKcHW/ARRPHj+myAqR7tgEXZbU9ZY0TaN5oMb5AbxwoqyC2XzGYr3g/bu30hEH8bHy4UemIEh6qo/TngEd4sUa6PF0aFosrVI45bFBtl1Kgw2BFNloGWWkUChkHAwKgubBPxy8UXgdor2JJLT6iC15jRSxeGHH/BS59pU6ANWftFKH51X8MIpqoh4cdHvxv/HTli4WjcMxiMXOjzLWBe/wbiBImiTBD8AYfaFEdDoVwSmMhugF9eeWr+qTLstoTTAmOnAGgtZx9AyC1Xm5cWRpilaKvh8OHZj5ZGvmvYtaMycZdWoK43Q0bctqtSZLc/pe6AhuDHHEqWnaFudlFPr7v/8HtNGi6meiMTiJ3SLQtpLDtv3wgbPTU+bzOXVdc3p6wnazYRxGri6vSJKEpmlYLBbRz9sz9P0hPVirQBH5WCJMdRRZTp5nNHcNg++ZzUuWyyV939F1DVU1Mo6KdvQ0Tcu8TGmblhcvnnG93zGfz0nTRMZ6K97deSYmfVVdk+cFw+jY7Xc8evwIYrz8fr9jX1UcnZxwenpKXs5ou46g4PXr1zRNy75uqOqWNLqMWqWZzefY+HilZaHx7Tff0LYtRmuyPMUmwjjf7Xc8fvyIqtnzL7/7Hbv9iFYJvfOMTpgnKt7Upw4qzyxFbkmsJ7FBziltWC4WLJZz8boylizPosxKOrCsGMjyBfOjE9Yn55ydnpHahK5rqXZ7bq6v2dzekyUpRZKhgyyCNpsN11eXHB+fsNtu2G3uKbKU2ayg3u/Y73eUZUHne7p+B0GR53MW5Yqz0zNMovj6939gt9+A91F5EKKZ4o9YpKYrQABqERo7BaPy9EAHNECDYghgg4xsWilSBb0KJAoSLeVCFNiy9hdCdLRg1QHvFaNyjFpM5ryKcHEE0okXrIqr/imgUU+SFyYpSwTj1Z/YtfBA7py6KhWLUIgUB3EAkIvIxQPro2+P8J7El8p5J8XIDwTfE8IIYSS4yOmZThZiYf1E6ByYOrXpfcirnoI0jdIErWQZE9/T6EYYo3ovdlZaa2yS4ONGSQqVwVpzMOl3ztEN/cHREg+73ZahHw9ZdpJdl5CmOU3bkaYpXd9zc3vL23dvOTk9ZbPd4r3nfnOPtQnL5QofAnd3t+x2O9qu4X6z4eTkhM39PfPFgrquSFxCmqUipzCGxWIRU1962rZGK83x0ZphqMQALrEoJXbExiwosozLrpOoqSTBMzLWPXXd0NaOYYTWiUxoPj+Wk9saylnBer2MtsOexBq6YUArTdt2fLy8ZHCe+/uNeEt1LdoYurYVTlSW8fjxE/pxZFft2Wzl9XfdwM3tPcVswfrohLZpo9/6HJsk1G1L1TTMZgWb+3vqeo/SkJcZZZ6igse7kdlsRlYWfPXNV2x2O7o+xxiEye3g0FkbWX4YpbEqENxAWQhx02jFvCgp8oK7uzv6oaMsMyF4xtScxWLOYmEwWclqfcz6aE1qDWPfMwwdXdMwdh1lmsrmN3jcKEv8arel61uqasO7ty2j6zk/PaPtMpq6ZlaWWGMZh5E0ESvpNE05OTlCG/jD7/+FutqilUAQoCTVWf/ILgjSkCh51SGu9yPu5EKgD47OQxUCIx4bt2mpMnELqHAq4GQhh1JWNly4Q9PjtQC4TitG7RhVEM9oGXwOWAyTdXCYLv3JakIoDD4Wp4d1nTrQFf70zxP28+nmCzhs8Fy09RX3R8GXrFX4RONGjfMKNzgCI5pRsAFk/AvBMfSOxGhMGuUAPPjCT8ENEhjjD3jNgWIBh4JzABl1LFZTt6dUHOfkwz+Mu3Kw0NE1Lk0z+lFoCT6EGOvk2Wy32FT8sbthILGKLM/I+hwziuNkmqZst1vGceT8/JztdiuJwtOCFcVut5cChuf69pa//7u/4/buVrRaWUpVN5TGHqQtRSFg7+RsEHAslwvq2uGcRKKPvWJzfy8WNAGsFXFs09Y0TcN219J2jjFA07UU8zlJYTk5PmJ5vKJtG46PhXBojfCyTCTxtnXHdrPl5vaOfVXTDT3Pnj/jw+Uljx8/wqYpzjnOji7w3rPf73BBOu6iKHlTvyMrCp6/eMnV1RXDeHPYyN7d3dL3PYvZjKPlirvtHSbALMk4XqwEuA+BJE1ZHq+5vLnjft8yeEM/Btmy6ckoU8TZaWIwOqBwpNZwtF6xnM+xxjArJEj25uaG7XZL0+zJi4zFYsbx8TFFXlBks4ibZZR5FgtUd1A++HEkzxLybCWb0XFg6AeauolWOb0sNba7A/k1+EQ2t9rgosleYjOMsZyenlE3O96/e0vbVkieYYoxeUzT/vPTxJ/7+t8VaRXUNPw8PL2KXZH3MATFEAJj7LZGtFAWlOBRk89UUB4fxHQrqLhpURqvNB751WnLqGF0sumTQmZQKnYzsduQ1yMn74RXhYkxOZWFAOK0GakJ0b53ohNMWMynxnbqAFh7Gd28E16IG6SI6IBODNoLqChCskigCP5BEuQcfQfaCC9GxY5PQUziDRIGGpnsQiWYwPSpgJpDsTEE3PjAd7LGxAy2/jDCOvzBqWGiWMzKBTaTkM6ARpuENIPGtdR1G6UiCQTRAdr42pbLBVVV09Y1H96956/+6q84PTrl6uoKm0iSsIhw5WZTFjOarmW733F2fk5dVSyXK+5u72jahjzLsYlYByfWgpcbjveO5WIOiIAWH+Lr0GJf0vVkeUbfddxcX9F1sNk5oGBX3xK8I000i9WCn335E06Oj3j37j1ZVhCcZ76ccbxeM47ClO+HAT+KGWHnBp6/+ozbm1vRKAaxUi6LGUZbWRp4z/mjJ3z8eMVXX32NNpqjpVzQ2+09RSGmkCKfqUhTy+lqxaqY8f277zldrymSjCLLyLQlzSUVZ7PZc3O3Y1cFap/Ijd8NKDegVMBqMFZRFCkqjPRtR5aUFFlOcIFqX+G6nhA2h6Rr52SLm2UFVicHc8RpYvDjSLXbgtJkecYYHG7oKIsoe6pr/DDSxsy8PM1YL48EAz1KOTo64mi1omlq+r4ieMjSNGKlmqdPn7Df77m+umQYZIuXJWn0TBMMuhtGhvGHSoj/w0VqAq4n8PoTKPgHILVc4tIxTaYin9q6uAik++lxisPS3aMjR0rjIljulcdNdhJxVAtKxkQTca/D+l4FQuQDTc87jXyHr3hxT5iT8KMeOpuH/yx2V4EIjo8MQ8Mw9NGjXMI1vTEoaxmdkkLl5V3riJcpDW7sGXuFTlMpfjE2W2byQHAqHhEnrzd2UhPJdCpQn76HyeZlyluzVjYqUyzS5L8OIqEpk5Q0ERlJPww47zFJSppKLHpQYi8yDr0QYK3FJgmz2Zyu7w+RU7///e/5t//2/xTv2jvW6zWb+62Y2vUjs4XwkN68+8Dzp8+4ubklz1JOz84xWrG5u5eOMWItu91OsMso6WkbccW02sagAGFf+zBiTErXN9ze3jIMhqbTB4/5opDt3fMnT2LceUYe9XB5mjGfzciyHOeqw9ICYFbk9GPPs8cXvPnuGx6dnR6WG+ujNfv9jl1Ts1ofsdvtubq+oqpEzFwUJe3trYis8/wQWipY3Cngub76SLXd8eWrzxibjiJPefLsCdc3N+z3e67ut2yqjrodZCmiPO4TWCEYjU0MqIDyI2VmOV4uKJKU4AfGEZzRElCa5nivMKahqYWZX6SSe5imEnYxDgNNvQelKMsZznjaRqxu+nagG3r6Tt6HMYqiKMjSVHzN5zOMNgIN9JI60zUdPhVW/3Kx4PmL51xeXtIPEsABOfgQ8UbFODrxfo8F9S/5+suLVBwRwtShKBO35p84Fkzre2SEGfD03jFow4D82XrR8ukAE2Xy00f6gBQpLF6D1xavpTPAT44DMjMKhCXjnYmOnyAFVYqWOUhGYjCaxANBPBE0k/XKQw81lVN5PmEliP9017SMY481YruigjggHoTWTlwdVRhRQUpuiPOX6zvG4AWD05pRKVwcQz5lnRstA+GUbmyUOmgRQby6TBxpp+JqogPk6Dx5Ju20JLe4WLAG0q4nz3Lm8zmDczR1S9N36EjtEL8pzThIAS9npeBnRjFfLtlttxilGAbNm+/fcPHoEds//hFtExbrNfb6Bus9PsjVta8r9nXFanXE/d0tPgTmZcl2uxXbXqUiCVWkMFoprNLUu734VWWygRu7HpwjSS3Nfke9r2iblqoaycsjYWw3hrLImRUF52fnZGlB1wlT3RrFPCYVx/oec/QURWbJjOaLF8/IreLiZE05W5BmJT4EdvuK0Tm6pscv4PV3r/HBk6Yp+32FNQk3d7ecn1+Ip1cvWsv1+kh8svKct6+/w2jFajHntml58uQJXdNSVxVuGLBaU1UVwwBpVqIzRbOvMNayWq+o6h0uSFbecZlzsphzsppjjdx8VusFRZYzDo6+m8bnkSQx4uZQV2LWFzlby/lSgHQN3nU0VUu1r+P17ajrmjwvKcsySow68mzBMPSimkigrRv2O3FjnaRI6/URZycnfP311yRZTpbmDH0jAIxWEWaY3GodU3juj1qknBPl/+gDzqnoiSTbhwPbJ3Ag7jlg9J7eKXoUvVZ0waOVI2hIeZCREB/vgsKHiY4g3ZBXTsbEONppND4K2QzirjA9w3QCHjZ4arJiUQeqw6f2wFJ/ZF17KHDhgbs0rfpRRB9oj3cDgwOCEXjOx7IdENfM0SEBCiLfUFGjNw4DKgS8MXhtCEj2muATU8Ktlpw5UWbL8dHRLX4C9aO1jOGhG7SRDKicO0R9A59o+Rxt25JY8fAp0hSRpcLQDqSJYFIgnCujNMerY6qmYYzOAM6N7O63DMPA/f09q9WKV69exZM6Z71e0172B37Rcrnk6uqGzz97xX63p6r2pEnKYrmkbyQKazYrhQ4ximJez2cMfY9NzEFWMo4jxhratuH66lp8iGyCcwKia6NZLOcsFgvyLGcxm4k5nxtJrRZPpSyL28+HcyJNU7LMspjlPHl0xtXNNalRFGlCmids9jV11xKUZgzCpWrbluVqhfeBru1pI7F1uVpye3PDxLI+Pj6ijxewc46LM9l2Zjah6zru7u95/Pgxg3vLx1txVVAqoSwK6qbCWAkwkLipwDgMXJwe8+rxOQkegtBHLh6d4dzIu3dvUFiszei7gbbpgIyb61v2+xrvHavlkqdPHst5EqGMrqvZV3uGfpCutqpAiZX1OA40TYvWlqZpGMeB2Xwm/vreMZvNWK/WpNZyfnZG2zR8++23LJdLipncENzQoxON1QZr5MabJgllWR6WXz9qkZIXJ+S30SvcAdsVZvnBVjd2NjoSD733eCUbv155tBakWOHR0dhO7G6RTV7QAiIrIj4lz+fDQ5dmDpu5H3o6xasa1AP/CaOjNYUQFdX0b9IiMZEk5XC5CGbLXSU4kVtMAaxiSSHFS0bQ2NF592DpOwotQWvBdnTkSY2jw09r9iSJhU8oFmrS1BmDsubgxuC9lw/zk89BKcHQ9HRMAigVyDORP+R5gTUJIUAXE3gTm+CCkDnBY4ImyXKWNsV7hXMDWluUkm656VtOzRnWWqr9njCfMytnAqK2DWPwvP3wnp/85AuatuPt+w8cn57ROUnlFSDfM3opqscnpzjvQBuSxGCNWHckWRr//Zi762s5XlqTGPHTGrpexLfpjMsPV+y2G+bzFQqFcyPr9QqMZtZK55PlOXW1Jwwd7W7D0XpB7xxj3zBaKxyiEFDBkaUJjdEcHa1xY0/f7skSRZ4lbHZb3n+85nZfYZJSeFNZKi4Mo2O9XjMMI9dX1xSzOW++f4f3njRJ2W42dFVNVe3wQ0c5n0UJSM28nPHh43uhRFhLU9eM/UBqU5JEpDFd33G0WtF3HUPbkRrF2dERn794AU2FSRVt1/Lo6ITlcsFutxMhdj6nqXs2ux1129MNvQQ5xGI/m81p2g7n7kgTSznLGfqOrpPN6m63o247lJaFSNhs6fqBR08ex5itJQrF6DzByc16XuY8fvyYq8srLq+uOD49Iy9yrE0I3pPMl1gjDrR+HEVHGE/iIWZM/qhFalJ0T8wk4p18QlLG+O0Qm1AjOy4e/Cnl70cvVAQfi5oLgU8dDiYWsI8bLxdg8HKB6xDX8sFzWDNGKoCO44oxBmPsg/xFG+F4oSMh9cHpYGJvBx+YKtGUtuvGET+MD8GfB5xIHibkSUAF/DActoBuHHGjRFYrRD9ntAQING1DiCREo7SIaZV0o5GZ8QCIWyv/Dj8A/FRUvqOkCz10U0pHD2kbXQUgqBZrLUVR4INn6KUzGdxIFjVwi4WP5vkeaxVFWXC3uef27o71asU+BO5ub5nP55yendG7kWpf0Y8jv/r1r/mHf/jXvHv/gdvbO15+9hmXlx8hBIZhxI2Bm9s7vvzii3h8OoYYPJoVYjYXlCIx4oogmyxDkiQHqoQ1hrLMWC7F7bPI51xd3zJ0UoDzUoiF6/WS1WrOfn+P6VvSMKKGhixJ8GOHYnbY5JZlibWW7UYxn8/wfsAaOD05Yd8OXN/ecXV3Q9WOnF+sODo6ptrtKMuSfhTpyGKxoG4aZrOSqqrFAC9aAX/7zTdkWcKsuKAsSoZxYLlY8vbtG54+ecxut+Pm9ppxGEiShGVSsml6qmrParkkyxKq/Q5NYFGUPDo95f7qmhTH2YvHzBdZzDjUtK1QH7x3tF1LVTeo2Kl778XbKnbS95sti1mJTUQOtd/vxa8+nrfjMOAZGPqetu1I0vwQzpokCfvd/oATLmYlL1++5P379+yrPScnxyRJKlrNVDzQJOLN0XftJ8nLkvc4jONBaP6jFikhb3okYyIKWeKsOTpHHzw9gVFFoqaCUSn6EBi14EAe4UGFID5RamJlHzo16UgGNxKcZ3Qjw9S2qU95t47JMEW6iwdXAxnvZMTTZtLp6YngFJ9KHa5+P3mpC5cd78eoyJcPT/LsPCZKekIsUj6a1o3jgBse4q/96HBhRCHGbDo6H1htHgphLC6RRRB/7tQVxpHOTiOdOhyfgNiPqNiBCu/EgeeTblGTpBlZkA4YJVlnWsnIboyl7x1KjSQ2Jc+lhXc+kNgEay1X11dipp+m7Pd7lNayzi4Khm4Q+5K7O7797lu+/OlPefP2LXlecPHoCZvNvXQJrqOqG+naVivubi9J0pSynGNMQlNJzh2pVGgZecGPDmstq8Wc3W6LNZaLi3MAsnTGdtcwny9YLuYUs5K2qTk/PcEmcH9zz1EhhUmbRKxZQscw1uSFeL8neUrfi3d7lhZkaUnTbGhqETfve4dJcxYJrOYFSWIZkhRlLDe37ynynL6XdJr9bsfJ6RkA33/3LbvtjvmspKlqlosFy/mcpmn48OE9CihnJe/ev4s+YYrlfEGxPKb+5nvwjtV8zub+DgOkacrzZ0/xQ4frWs7Pjlku5ngthfLdu48Hf3atBSOezwQ/EntqT5amIrrue/q+Qy8XuNGxbTekiaXIMoZ+IM8y8rxkGEfqpiFNE05OT2LIRcfrq0vc6Fkt11ycn3Fxfs7r16/lGEanVEmqtmR5ineerm1pqloSpkfBa9umlRRlNzIOP/J2Txg+/qHbCREzIjA4T+88Q3D0OjAEpIoSo33w5EGToIQciRdT9thFiY+SMJ48HhecgMxB4sfHuLaPl3J0DTCIpk7AbROLj1ZSlFBiyTKxy6fOaip2UiGmAVKeNyBjnht6whjvLmOPdxLw6Kd47+AhkjVDCPixww0dfuhxYx8f1wnDXkuckbUG7xMCE5Y3MZk4YGQ+8mJ0ZIwLsKgO27px9DgvBNIDluc9oe9BBYwV54HJv92YBJToDpXWpJmlaWISSXR+DEjk++gGhlG8u4+Pjnj79h2bzT3L5RLvC9qmYbvZSJxTiMZ61vD99685PTtjfXxM03ZyYlvDLkkpS0dezmi6jtliwdu335GkKSFAYtN4s5H3NQ7uB13srJSATO88Yz/QDy1aK05OThidIklLHj++iM6wjrzIaNs7tB6wVtwrjo/mXN/fschTVCiEaBuPetOI1ixLUoahijHxmuvbK7bdQFbMOFnNOV4u2O0arNGsj46432xYLFdcXV5J1zwMZFnKhw8fojNlwpPHj/n44R1f/uQnbLcbvv3uG3Ajf/3Xf83Hy0txClAS9vD05Jxf/f5rxr7jeL2ibSrGoacscvI0odptCX3D8UJ0fkmScH13T1VV5EXJarXG2oTN/Y6AyKN2u4okicx2a3FeYIYkSdnvd9SVwlqFmckiw43im5XnJfv9PpKYA6enx5ICU9VsNxuOj4/54iefM44j//iP/0gaOXRFUcTzVRZRdbUXk762Zeh6/BgOEfT90Mfp6U/kYj9OkWK6pGIXFcS2I5IQJfZIfmjvPaMLtEoxWg06UAZDOZ2E+DjKRbcAhYDKQYihLsho6Lxn8MI2nzhRIRIxxbUxCG/KqGjRGw6JvkaLP9TUXZnJL+qT9fNhpxhC1BEJqc05kWk4JyGb49iJAVg/CN5EjKbyMleHsRffo67FDz1+HGJY5IBNUiyI5swGRi9aLI+MbuIDH0mXsVhpYzFWvomcL+VBW/HhkkDSiKI5oS2Mo5PHGIuKHCftAjYVJ4SAdFA2TWi7jizLsdYeAOUsz3D1wGZzL/QDY7i5vmE+EzzKjU60eHkhF3Y0f0Mpuq7h7OyC+80WpTTr9QkEi01S6qrm7dsPvPzsGV3vsUbhXcNysSJJEvpxlPV338du8YE42LYtxhi2ux1Vvef4SGxSitmMfLYELRvCfBwiu75jMS/IM80yP6LraskvHDoSqxmGFq1TfHCMruPoZE612ROCjF2enrptKNKCXVVzvDTiLd/3pEki7gzORSHxnhdPLnj67BnfvXnH5v6e5azg7OwzijzjP//P/z1Hx8f8+je/pm1bnj97wovPXvKHP/wRrxTb/Z5+6Nlut6SJ4cXLZ3y8vqKpK7QKZKnGWkVd71mVBX3cVu53Wy4/fiTLS4q85OjohO1mC0ThdN/JEqFIybOUuhFdZnCej/UVWWrJ04Q0tTRVxTgOrNdi9QyCae6rhqdPn9J3HQS50T569Ih/+Id/YHu/5euvvzmQcIuiiLZAcrPoOkdd76nrOjrY+ojH+tj9KyEPh+THB84/JXASQdtx6giCxwREvBrA+RAlCo7OBVRiWCNkzil6eApUkO27dCfBO0mj8dMd1QnNAH2wPBGvPRGJqDClwfxwq6enb21kra8evMsfyuwDadL7mJ6B8KG8G+P4M+DCgIvAuAtjdDiQxxmt6byj61ra/Y4wdlgEcPfeCQYV3RN0YdE2wXgl42HvDmxyYxKxFk4SbOygtEnQNonOEzridA4TO1lh7SuMtoDG2EiInY6R1lKgrLDGJX7KkWaZ4AHeMwxyEno/Mk9LyXm7lnj4LMu4vbnh8vKSx0+eSEFpW9wwUOQpWUjQ2oiX2CAF5vj4OK7wC7o+bkyrls22xo+WxeKC/faW0ffc3+8wSpJ/szyNgRLy6UwJwVobbJIwDCOnZ094+dlndN3Avr1mfXyKQ5GVOcUwsChz/FhifEuWKvI8pa725HnBarEmMQnKpHSDw409xsDQ7zFJJ86wDqqmZj4raDcD7/c1lzd7jpcrFnO4udtws6sIStN2PeB59uQRRZ7y9s33DH3H6cvnfPHFK3b7Pf/wr/+B71+/ppwV/GT9OXVdo62lmM3o+pGAGOYNQ8d6taQee4IfKXIJC/Vjj7bicFkUGaVREET5sFyumM0XEiW2b9jv62jBPHJ8chSbRc9ut2O338r1EgJpOrHDA23bxQ10xFd9kBu9UnzxhTh03t7fYxNLWZQ8f/6cj5cfuby8YhZTnYEoaJbPrK6reMNpDw6kzjsIQg5ObUaWFaRJchD0/8hFKuZ1wUH0O7GJJi7Qg7skhwLUDiN7N9DkgU4Z8gCJkm5J+RAN8hzKu4MoN/hplR/HmjDRFTjQFlTgsBWT4hSwWkfZw4N42Ezg+YGc+WkXNRE4AyE4Rj8yjgPDOBDGkTEWGxnRpnAIEflq5T/xzYqYzihi4+BcLHJiDif+TYaiKDCJlkIcGeaj9xKBlIuxv4SBJliboW2CtlFC7TzOafGHVuJAYZRG2/ghKo0y5gD0ozRJKpFNWVGQFTl3d/ek1pAoyedr6oa26ej6nrD3LJcL1kdH7HY7AMrIazLWcnpygl4sJS5di0Hg6BxlmZMkKaMXIXBRzlFasz4+oaorZoslm82Gq5sbzs4vcK6nq3dU+5o8Tei7jpPjNVdRATAFNswXCxnlAgTdc/7oKcVsxRD25LMFq5MTnJeL6vT0CD90JNayLteUSaCrtvT9yGy+kCCEoCVgFccwdOy2G6wNpNZQNx19L2Z5eZ6TtSLYruqW/b4mNYqmaajqmmK24vvvv+fFkyfU1Y7tfgMEzs/Po+i5hxCoqorLy49ijEcgK3LqtqHtOu6290KezTLapiMrUlSwZNuJXuKjNiKQWMNqNWeZpWgv4aePLi5Yro94//4j11fX2CShbVsBtSPOE0JgX+3RWgTlqU1Ik4Rh6EVjl9o4dsqNLqCi53zKbDZjv9+zmM85Oj5iuVxxc33NfreXczh26pPDRtuJvEZ82x277ZZ+6NFBNHplOaMoZhR5QWoj3eWhVPx4RSp4RfBaCkjgEPAnjoL+oBfzwUkHNEiLGfB0aqQxikYZclR0FvRi+TKFFDpJR3Fe1tfBiQTcjz5KJ4Q95YPHEUgif0jHjY2wDTSJtZEV++Ab9aeG79PE9wMTuxAiWC6FZnIPCF4cNMPh230ibPYYLdup0Vg8CufEq102gCOD6wkosrIgKwtxwdSJOCYcjplIX5I0x6YZWhnQFpTB2FSM3oxDjQ6UmOu7vidEn2qNxiQJSZJFwW4Xo5yMuFU6T54lZHkum0QjP+/45Iy+77i7v6VpGgbnKMo5aInDbjPxud5t92SprJu32y0hBLIs4+7ulrIoKYocmyTc3m8o5ktSW5CkGQtjscrivaeqao6P1qzWR2zcyCxPUd7RNDIWaCU6PZskzJIFR6fHDIPDB0XpPGePnqCTjMRpjtMZs+URWfQE1wp2mxtWiwWldTDs6dsGqw2zcibLgtTi46KmzHOGLqUoDPe3N1iVgpfuBjxJkpJYQ54l3Nzc8vL5Ex4/fUozvqVuO9w4kljDMHQMzvHlz75kVi45Wi753W9/zYsXL3j93Ws2W+GVWWP54osvqPYNwzhyd3vH82cvRJwXYueYGLJEJE5WiZA4NYb1csGinNFVW3Ir7yHLCr799lu++uobjtZHjM7Td1JghjBgrIQeHB0dsZzNJFDCGu7v7ths7jk+OmJWFIxjT5paytmcrulJs4SzszO6TpKzX754QT8M/OF3v8M5R1VVKJMwny3wBLq+YxjkZh58IEkydtuKu9uNiM8XS87Pz1gsFoCn7zoZAw+2Qz/yuBdiYssYAoMPMjI4j/Mh4kqBMTgGN9D3I2MfcKMnKMegHZ11tMrTAalEqsr44jwquEORGqNRG85DcLhhkCwCgbUBL/FPiXCFTFzJm4M8JBEWd2TVTl3Un31PxE7KT06bMtZ5J9tK0XYJv8O7hzAIebAc5OnnJkmC600U+kYGudFYZcSD/MC1ehAMjy7aCUeSpo1ky0/0xSiiYjxYnPVkQYSpfrcTEF8rUpvhfCDLswPYfnd3y+ADaWoZxopxTIWeEDdB1b7GJgmr1QptNPf3dygCaS7aupPjY3abrQC0eS64gveR6Cfbt9OzM4ZxpJjN4yYtY7evOUokrr0oxdq3LEq6tsH7wNH6iL7akyWGxGiGvsWNI4v5gr6rmc1nKByzUuxTnIesmJOUS4JJWOeynZrNCowK1G6UizvLKPSSbvsR19ZoHWRBkKeyEcUSsGg12dUkeDfEbrbn5uaGLMvxSnG720DsLG6rluPjFcv1EW4c2W23LJdL8BJDf7vd8OLzp7TtQGItd3cbXr1KuL6+oYtxXEfHR5yentL3Pa9fv46dtXDBlsslTdexbxtwnqHref7sOUfrFbvNliLLqKo9fVVRHh0RCNxv7rm9vZXPwo3g/AF3TJIkjmgrHj9+zGJecnd7x/XNNVVVcbQ+Yrlc0LVtZJiPDMMly+WS5y+ecXNzg01zXr16xc3NDdfX14dFSZblJFkurhpell0QyLOcLEvZbHbc392zmC15+uw552fneC8WOHW9E2lZP9LHzsv/2EXKjYL4j17uzMPoJB3mwHcSx4DRjfhxwI9yrYqyQqOCYXSOlpEkaEDGIuuE1OndQHCOMYwCYntPGEcYZfyTC3nyKScSQie73Qeu1DTOmU9IksAPC9XURcVRzx/4UTKb+xhp7Z10hS6SE8PkNxuLkA7S4QVrIc9RbqDFRw9vS5Kl6FjAprAEj2wbbZaiRiMESy14lbVW7HDHyZJGSJ0S5GjkeACZSfBBiXmcC2TWQBB6aZKlMl7kBW3XR11bLh5TsdXWxtC0HdvdjjwryLOC+axnGHvKohAuU5ZjtWU2m4uRmROjskmj5r1IZ+w4Mp8vcRi0TdlVNXXbcbQSQepqtYqSG0M/OhZlznp9RFttsdaQpSnjMLBaragryFMwRlb1QRlGbygWa7JyDsow9CNZdLXs25qyKLAqQJ5zt7/Camj7Fm0C2kRSGwprMrQt8Ba8a6mi8VpwnqZu0cYy9gEfRPxaZBmJsXQh4By8/u57qqom+JGzkzWzeYFNFH/3d7/g69fvOD4+5fd//APz5YKb21sRYBvF2ekFn796hTZa4uVtynK5oh9GlvM5eV7wzbff0FR7loslF+fnfP7qM66vLlkuChKjubm6ZDWfkRYpaZYyupGiKAghCMUjSbGpZTFfsFwsBOg+v0ApxXevv2O32+GGkSITGdLtzR0uOKwx4oxaznj05Cm//PVvePL4MZ+/eMHbt+9ompo0zUjTHJtarE3ph5F+6AlBUoqP1kckSSLE0THw6rPPuTi7IE1Tbq9v+fjxPePYRxfZyVfNP1yvP2aRGkNgCIE+eAGLnXz33tP7IBQEPzC4IZIfhcVkQyAPmtSDCaKL817wHzOOKBfwYRQcx42xSI3oCGZPILXgRpJWo1SMJw/6MOpppTFq8mqSQmC0+l8bzP0JLuWnwhT8wS5lImX6UYrUOEpBDbHLCmEUT4Ug1FZrDTrPMZFJ35vYFWWpYEx2ir4W14A0lZNNLnyHTVOMFSDaGEmqVRH308ZAUIeZf3p/NjKo266NHlWKtu3IswytNOv1mn4YRUhsxN2yyEuUMgde28ePl2w2G9brJVme4+qRfhiYlTOCc2RZyjgKD2e5LCM4WsfAhpbRjZydn5NkOfNywTB6dJoT4p1dAjvzmNEogadu9KzXR9z2Qikosoym9RRFTtfuSVJDkSekaUY+WzIqi0lnEqgZDKMRwTExwSdNU8LQYZREinfNANFML00LkiRHJTmJTTBpRt+3BA+JlaI0xkDSrt2R5kvefbwkjb7t4zDw5MkTdvuK9+8/oo2w8nebe/6Tf/W3KBxeax49fsSb798eivf3b94wDgNFnrFarRlGxz/+j/8jfdcTguL+fkOW56TZCZvNhs39hsGN5CbleH3Em+9es9vd8/jigs3dPXmacnx8zNHRGh9G+mrg6OiINE2ZLxbs9tXBAeHu7o6yLHjz5k10Sn2P1lpcNPUD+XM2m6G04uLigufPn/Pdd9/xxc9+xvnpKb/7w+/JkjRKhzLSNCNETaDzTmLti5yynGOtpa5rUlvy8589YTGfsd9u+d3vfsd+txU8WQVcTDzSkaD7F9an/31FagjQB4mt6gIPhSp4OucZgqi3XfBxnR8wAXIPS51QjCM2SodxXtwrncO7gA6yUXPOS8GKQLVyY3TAFIM8pXzsoqyIjUNkkhM1euphk2eiZk/H6g2fjHcRgxJ/cgHpwyDjnWBio4x7kRdF9I6exsLgRmF3Iyk3WivxKsoLoUH0VuKfsoysEDJgmOLpveTOpWmCtak4L0aTMG0TPErcCmKsV4g1NYRROtlYqLz3aGuxPiEANpF0YOecdExJQp5kwMSOj5hX7KayfIZziqvLS4pyJiECg6NtGva+YbGcEUaHtgn96PBVQ1mWLNMMAthGrHGtzSiKGeV8SVDiXNC2bcS+EhIM87kUaNe1dE3NbF6KgNkPpHqGD448S1nMZxjj5fXkM7JshtUJKilEthMUWZqRpAldW5PYhLZuCH1L1+5wY0vXVuAcWVEwK4/I8iUqKVGpBjWiNYzekSaWyglepHVClhXc7Ws+Xl6RlmvBNo3hfrPh7uaGk7MztDG07xvm8xKjDPPFguu4FNhXDdW25vLyPeenJ5wcH3NyckLTtNxc33NzLXFUXddik4SqarBJys2tRNsniSVJNHc3V+x298zLkvvbG4o8Y7may+cbXUzzLKdtO9brI7q+o25aQoD7+3vGoWe3TSiyTLp378nzIgrXYblccnp2iikyBhV4+uQx4zjyD//6X/Pxw3v+h//xfyLPUo5Wy0M46jiOtH2PUpLTN1/OSbMCoxMIgfU6J89SfPBcfvjAd99+S1Pt0VYd7HC0MnJMP80S+LE7qS4EWh9oQ6ALQTzN8XTBM+BxWvg82kvwgvKeFM06TTiylhLBU4yXqKepUwoHHEh+H8IUiCnguR96+b3iEAZhMBg8RvkDyGjiZk/4UAKcHxjoTBe6jx7mICNbxKJiqsWUW+emUS8WqT/9Dn5iyI+44CUJwyZoY0jzPLotaEySxk1jFCOHIBYtVgpUmmYYm8hoGOkH2sjoh9ZoHSbGBmiFiakmfS+apySZTOmSaEMiYQJBaZTRuChs1lpG7brpKAoB22fljKdPn0oCbdcQAqxiconRhsRmeO1IvRynru3Y7fccHR2J9CFLadoWm+YU+Uwu9FICF5S2cb2tUVaTRQ3jaBXB9wxjR1Hm+B6sSnCMKK1YH68PMVKJzUnSGUUxRycSGKGiv7u2GucScLJA6fyIUp4iS2iCJ0kTimJJWaxIbM62bTFBYdPAfrcjMQJNGG0YRymqs9mC/bAXAXaRc3Rywt3dPV9/9RU+CIaUFxkvXjwjiYLnu+09IUn5/t0H2qqjrTpC0FIUjOHp0ycS1HB3R9201E3LMHpOlzOycsavfvt76v1eipQ1+KFDRa2b0TCfzXj8+IKyKLj8+JFxHEhTy83dHeujIxarBeOdwxhDVddR9B9o2oayLFBasVythNDpnGQN5gVpmhOShBefPSPPM7qu4f/1P/x3mN6xKEphwecFPgSqOnp5lSWr5UKsW7JCSMCZ+GM1dc12c8vl1SUfP7ynaRqZgEbp+pPMythoU8GL42Y+scmPXKT8SOc9bQg03tEET68CrfdibhcZpEmAQmmwirkyrKylJJAFTxo8SVDYEGkE3h86G6KT1FQ88EJLECKVROuAECC1H9Cxk5LImSnJeJKUfOJlfhj3Dvxumfhi0EMITtjWQ8/oeumeRumknBujLEboECFuY4IXQqUKI0JfGQAfZS9EQqVIcg7WxFqBF0mPjp48SZqQFSVaR96I0ShrP8kPVAd6mlGaNI+s/MkdwAiPSO7CMiJlmcTcKxQkGh/X3AePqYiv9Ylsg07Pz7m9uWJ0jjRLWC6X1FV9uCiXSwmznDRX796949WrV5SzGWmWU9cV58YKXysoidTWEvgZlEIFhzaavEjo1MBiXjK0tby+1KKVp1S5BFcYfXBq0Doly+dk5RKvDdFbhNEJv0xb2YDaJKH1jixLwFuxHlaG1fIIpTR1U9ENjkRb0Iaq2nO8ngEwugEVoNrWtHhef3/Hrh55+uoIawyXH99jrIyWy/WC9WrNt99+Q5bMJKl5v8fM5uzrjrdv3pNoSxh6mrqlLAratmGz2VDVDXXXUXcD5XwOSc6vfvtHvB95+uiCs/NzZllCmSdstvf4URKH10crNvcbtvcb7u/vOL+44PLqmtXRmnI+p247bu/v2VcVBFgslvSdbHaTNMM5x2K+ICCkY53EGG1j+Pyzz5mfrPn2u6/55S//A0WWsVotmBUlaXSh6MdGtq3LhZB4szz6dAnDvO97rq4u2WzuqfY79vs9NjGkIcGNI3leRC1fhtGWsRcMUCtNlmXiZf9jFqlm6Gl9oA/Qu8BAwEVP8RBrgEFTaENqxQezRDEjkIaRHE9uNIlWWAchSHCAcKAky35iCoSg8IPUlKBkw2W0JLWmk1e6EqsWMXaZhL/qIfnlk7TeafsIUSuohLnumII4o/tm9LvxzsdOaoxERScSHufwo3BRxmFAB0+aRl+tcRQv9siQn9wPFPKabHQviA7EooEMgSRJMUkWxcXC75oY9g9f8gejxZKlCGLf8WB7LAdKa4NNE4mm0gathWiXpUKu01YKYZKkhyKYZhnz5RLnnaSbDHJybTf3ZFnGfDbn+OiYm5sbCap8/Zqr21vOz88wSUquEm5v7zh7LDYeCk05m2GsZejHaBcj/kGJNVgSEjLGPoAJoEYyI1IZoy2JzUizjBAMNi1QRnIDtdaMxOWGFlZ+iGkjWkGapNR7wauClwCQwffU7YhOcqa4r6OjI9pGTPratsW7gFKW199/5Otvr2iGkZu7W3EN1fD85TM+Xn7g48f3ODdycXHBx/fvhaCb5Qyj5/L6jq739L4ji110WRZcX19zdyfRVM9ffs4fv/mWcrXm3eUN22ZguShp+p6fvHzO9uYjic15+ewp7z+8ZzYr2e92aCXcOGstN9c3eAXn548Y3MjV9TVN24nsK0rB8qKI53dgdCM6sTI5OEjzjDyf8fOf/RXlrOSX/8t/4OPHDzxaHhFCwGpDtd/TaDg+PiazGTZNSVKR46RZSlGWgkm2LfvdXkihSuADdMIwdCyUYjYrKEsRMg+DyJuGwWHThDwVfpbzP/J2r3NOwHG0cKTCgeYBiOtBglABvJECUvpAGgIZnsxCoSH1ghuhrST6jpJbZ+HgrxCiREaELhYVHEZJXHtmDLmxZNaSGLEgVip2DmryjxK3Az1t+AI8zE0yDz8kBU9j5gNgPkaF9hiDArxzUXskI+gwOPquFz1fMKQmJs0cxk51WNkTRLoy6skbasr/U7R9TzYMzLJCaAbGiiMDsVhFHlWYmP0EtJYIqNGK549sHz2jjttHJyOtmOFLCMTcJqKZ8kEM0AoZDyVw1JO7kuC92AEnEfcoSoahF6InivXRCeVsTpJmtH2HCxK7tZrPGAbZiGa5RJgnypKlKQpF34/isxWkCDo/YBKLCcKfAo3Tkt9nkxxr5YJwKMLUQWkVx/1ooBYLs1IG7zxplsJYk+cpt33Losxou5qhDwzBMisXaCsRX0YVbDdXtK2In60uqYeBUSl8Ymj6nnfX9+Ad9/d3fPGTn1A3HaD46U9/KuzqtmU2X1D1A5tdDUFFa+uATTTLxSKSKw2PHz8myWdUdctieczt/Y6PN7dYm1A3HYkKbLc7gvOSwFLmWKMOCwsVkK1rWVLt9zx6+gQXPO8/fuDu5i5qQj1JJvY/U+qP1rK8aNqGi4tzxnFJkiT8/d//AucC//wf/pm7u1sZn5ViGB1XN0JrePrk0cGaJgyDaHCN5Gz2fc8wOLbbXVRhaEavCMoyX5TMZjOWy6VgWUNP3YiGD8TjvK0b2r5jjPjpj1qkfNARFwClPQwhcqfE21vKSTQM1gETPAmeVEGuobBQakXig0REB2k7gxG9nHZBUjDURBwVqwelDRHtwipNqg2ZsWSJxlrZWqRpcsCg1CcFapL5SVfyYBI3AdhhIo2GGNXlxR/aOSfd0ijbMTfK9lEj8gLvAl034IaOvlfYKAm02pJnGWmWCFM9uHjSRG1itF7Jy4KinIHS1FWFTVKUKtBBY6YLEy21zD/Yx4QY1yWeXdEkUGm0jqaBo0h4rLEEo/BqoO2HgwPi5IAgm0ehJYTgMVZa96apwDvyPMdozX63o6qaqVcly3IuLs5pOrFcSRKROWRFIs+tRPrgXJDOKRXS6hCXHzLSx5TbQ9gqWG1QJsUmGUkiPK3UWMHWtD50ldIdRhwxQgY6ulv0Y8fQVgxdjV6k1M0Ob8GkS5I0xSQpENjuNnT9wO3NHcFDF0Zuq4oe2HYNrR95f71BBSETf/3d9yRZzsXjp7x8+Yp/+p//Z5q6JitKgtJcXd+y31Y4J69VwPBbnjw6k/NWW1brY77/cEm5WPK7r79DmfSQW+i8ZBK+OD9hsVigcTx9+oRqX8vWNgTyXPDLi4sLglK8efOOu809282WxCY8ffqI8/NzXn8rdAPvZYP8+Mkjnjx+QlGUdF3P3//9L3j9+lu+/fZrjDGUeYYqclTwbKsNy+WSx48fc7RaxpsTbPd7lGrRKDbuDh0JpVonYgmjNOvjU1brNWUhJobT+Tp0rWzGPfR9e4BKpvNpHH/kIhWm9T5RNweRNxQ1dBFXIroEmCBZe7k15Jr4rUhMwMbRkNEQlbPoUWQeRK9r8VNXkQ9l5DmBVFuyxJIYg7WSOyf+TCKKnYzTiMVJR92SV5O3eexKIu3g0+j2iX4wRr7XwQMnktkma5lhGBidY+gHqqqT4AWtDkUqy1KMVfR9F0fhQPDjIdUlyzNxFzh1shlMEunQrPiQJ1l+GA2FGvHD4ioiXCHFoRSJtXgnjF/tBRROs4SmaQHk5DeSqWeMYeqyJ4mPJKAUaK2p9lvQluPTFdpYttsNKEPVdGQ+MJ/PsdH4sJgtSNOMcZRRtutatJVkHIn/kveGHxhG6IcOG2LRP5ygGm0s1uYYm2GSFBPTa9xUhBSHcW3C1uRsC1hrqKqWptrRbjYUeU7X9Qf306woJA8vsdRNS9s21HUtZ7I23Nzc0TvHmw/XbHY1aT7DjZ55UaCVZ7Us+emXXzAvC66urnDeUZYzqqqj8YHryxvSJDlsjhObcH52TpFnGKsZnOfmdkOel/z+668J8dycrIqm93Vze8tqUbCY5VxfX+HGwIsXL9FaUTcNXdtRVzU3d7dkZUmeZYTFgsePH7NazHn75g377VaCeLOMo5MTFvMlfe9YzDOevnrBf//f/3/oupqm2TObzTC5RiEypKdPnzKblYyj4/X339N1HZvdlizPmc8Xci4HhetarEmZzwsWyxXlbMFssRIsUqlD5H3XChu97wf2VUVV7ah3O9q6FkWKf/Dg/9GK1OSQGRSYEMHp0ZOEaHEyCXGC4FEWRaE0hTJkiF2wVWAjpmQFQkYFg8JKEKEbmAIXQpKCFu6UAM7ii55a6aCM1ViToM0n271oUQI+bvYEr5gwIhCKBGGyiZhEMQGPwwcnd1Bi9JQS14Ixcrgm36veCWDmPXTNgPP9QcSZxE1dlmcPFAZEhNt3HT6IRKUfZKOlkPchHC8R3PajoyhKMDoWWrFMlmAGuVNNnCrnPX0/oI1Epy/XayDQDwOFsvRdj0KzmC2lbbeW0Y1UTc1isQSMRBdVYkdSlHN22x1KG07OzrGp0BjmScJ2u6XuehbLNX5XEVCxYzJAlOCMA9pIRzgOIpqV9BzJ0Bu9Q4foha5EAK5UirXpRMtFRbRRRrt4UavomDGp2rzHKo0j0DcVeuxJx5EB2WqmeY63KeVsKQ4P3kG0anFOsJHRyfjS9CNN77FkpEFMFXMryUQ//+lPOD89w/vA3d09T58+4z/c/JIkKySsdBADuN4NOCc+aCaxfPzwkfPzU77+7g2tgyFo2mHAqfBgcogU3LZpsIl0rmcnK6rdntOTU/Ajm23FfLkU2Uma8fHqkmWWMpuVPPmrv+L45IRf/+pX+MExK+ecHJ8QlGJ1dMLx6RmvPnvF3d0d/81/+99grZbgUCVTxN3tHX0vXmhZKu4Qk6rAuRFQzGY2Oh44QujRNqGYF8xmM2azOcV8hrUapTxudLRdQ1N33N3ecnN5ye3NFftqhxt7VFSLaIRz1bXdj1ykQrxQEAfMDCtSmFEdsBMQ3MYEg/GePChSNLlRZCrI+Kd1LFI2Ov3K40yQDm3qGlQU5oWDc4IEG6Qmyl9i+KWJ3yI/sRgjvCWjlWjUUOAlhWNyPTj8L0QjuWn8C34qtQf/KemiRJuko73uGLd/wyjpGl3fRSzJC+9GaWzSkMTVr5/Y+JHU2A+OMShskqJNijEpShlmOpoJthK5lBcFNo20hCiYDoEI3uko9CRiaDGleBjFITFLcEZGP++9uEgulhQz2Wz1fc84OpIkZblckqYpdb0nIFa5Xd+zrzvycs6HD++Zz2ecnl2w3W0JQXF+fsa+qnHIRs4YkWWoGOsl7Y/Hjz2+b6BvUUOHGwbx19I2bmUNyojFsVdSRJWJIa/RuFCWK/ogJM+zDNe3WKDvGvTYg+tRYYh8II0xBbPl8WHLFVCH7EDnRoqy5P6uZeg9u23LfHHE4CpC1xHQpErx4rMXFGnCdnPH9dUtJyfH5KfCfeoGzdfffI9zIgL2oTvc/OqqwrmR7779FrTIh757+xE/CqyQaIsfR2xiyLKUJE1ZzHOyNKWua549fcZqtWK/33NxcUHdNFRVRREkdAPk16ppef+bf+Hd+4+4MbCczfnj19/y5OkT/v6zl3z2+U/49a9/xT/90z+RJZLis16vWS0lmPXqSkz6iixnu9kIXy06tmZZRp7nMsIZy2azJS9yZvPl4XoZx4Ghaxn6lmEzHixZRLheYa3h6PiUo+NjyUz0jr5taJqGruswSf3jFikhacrqXmvIjMECIw8FSiMYinEOgrgd5AZSJXhVQiDx4p+UKBUtfcVTWRuivmpC4yd2OnEzp2PKh8FqEy1ODCZm0ukpmipugg5bL+Ljw8REVwdawMSd8l7az/gXB+Gxip2C1hpHlMZA3EyIZ9boJRJqGEQHpo2QAE0vAQKiw5vcOOUn+wChdWRVT5q3aL0TIqSYTAlW0onRfzErSTPxjTbGEFAHZvwYiaGz+YIsz2maBh8C2qbMZzMCPgqap/RkcxjxEptG21kYRvFIyssZfugp8oLNdsu+rjk+Pub4+Ji7uzuMEYrCGPlYaSrYio6dLSjxeB8dKBn7bfDiKtHVqKHDdz2DtthUuER6ci61VrhlsUj5uA2V1+gxsfdWQUTdZZYS2hbf7TG+I4wNhA6UxnuFNSXWZOyrGm0MxsjNNbgRaxM+friiqvaMo+budk8oEsosIZ+l1E3NxcUFpyenHK3XuNFze3PP+fkF+31NVTUMvhP2tTGSKad7MIr1as1uu8UPLY8uLrjdVfRtJ1FdVcPnX3zJ5n5DVe159ugRsxR8syFNxUHh+vKKp8+eUlUV2miury5x3pMmCa+/+xabpfR9z1dffyNk4LhoOTo64W63J58t+bf/6b/j5OKUX//LP/OHP/yOk1PRASY2JUtTbm5uubq8ZL/bsVwsSayl7zqMNQey8GKxYD6bUdU1u/1etJl5TpkXpFZInOM40HUSTtsPQ6TXiL2MXS/Rq/VBjlbXFdvtBmUcaa6wWUaaFz9ukfJKHZTLCkUSs9xTZeLfRaxJP+jkNIHcKKwfscrHeM6HmE6rp7tjdPVVD7wp8UuSIjXp9gxxQxOdLm1kBUuxUpEC8ol98CeSmCnZTsWipSZP86mATAB17LCIr8MYcTjsvWfoOyambACCJsbNS6LI4CUbTTtPmhn0ZPIQ4nuIBc4qhetG2FR4LP0g21LxKRJgPi9yilnJGByFk4shsYl4gYeJixWEhe4cvulo2x5tPEp3FKVsWsQlIXLDJuTQh9h1DYyRIhCCuCM45+m6hrwo2Ww3fPzwgUcXZ5yfX3B3f0dQgfl8weBGkiRD+YlFD3meRS1cwxhkrDNuhLHHDS1+aHFjD0oSbJ1PyW0iOYqivCZYI8eVB3mTip72EoOk8G7AqkDTVfT1FoPDhQEYUSTM5wt0DMVshob5bC4XvVK0TUNiEna7PR64utrQdZ6r2zcYDa9ePGW7FWzt2+9eU+Qpi/mCp0+fkKYZb9++RWtFVTX0rqPICoaxo2krjtYnlEXO/vYjTx6dMvpRCs3dLVmS8vLJM3JleHt/T1kWrGcFyjXsuoYsMeL93dW0f2y4ePyY/X7H8fExTdPyzTdfkxUlJycnfPvda3yAZy9e8vr1W/Isp5wvePHyc7784ktsAr/81T+BCpycrnn3/j1t09K3Dfvdlv2+oqlrijwnKLjfbEgSy2q9RGvNPMZ/XV5dHXA2m2YymrYtaSJ/Hsfx4HqbJumBo3iwP5putm0buVupODQETz906P5HDmLwWh+2YHrqqFQci4JgRoaA9WLsK32MwypPEn/QtGnTWh3yhWFikSuMMge5Sog6H22sFC0lomETeVDTmKftZMuiYxf1Q4O7afaX1ytiYv3Jvx2stxRMGNVErDgQQ7VBFRmBwND3hxd+GHPjQybS6Ogdvu9IkgyCdD6H53TgvAijx7FhGLzYs273pGnCrBTnyzRLKLuS3vXMZnNsEiOtjRFGdpLFv4u41NCjtCVN86jw97TdgNHiKiFe8DpykITE2nedbCndIMfOKPCOu1uJR1/O53y4/MDm7pqf/vznPHp0IUZ+QZJZFIoQxINLEmyFaJnmlqHzuG7ADy0EcZNw4yB8NOXQIcVrS7AKZQ1jkM83xFTMg7d7CHHbyWGcHGPCb72/w7sawyj0imhdk+Y5Iwo3DJR5ISiE89zf3BCcqN6L2Ypv33zk/d2e+7rCWs3p8YLEOlZrsTy+ubnjVd9zkqZ0dxuub2747R//yO1mx65qSazgcW3XUlp4cbGCsWG5WnByeib4XDqQlXPOLp7y9u1Hvv76G6z2uH7Hm29/z8XJks+eP+Hp08dcf/xA5XuOj49ZLOYS7e49u/2O+XzOfLni/fv3OO/5T/7Nf4JzgeWi5NGjx/zt3/6C9eqIm5sb/sMvf0meB4oswZqE85MTrt01Hz9eotHkWc7Zmby+29tbxmHkydFjjo9PhMSb5wd6QF3X8pm3DW0nGJwxWrZ8QbiESZp8slU3ODew2W64v9tS16IVzIsSjWIc5VpwwRPUj5y7NzlEyl1Nxc4GGd3QGBWkm/Ee670QClEYJRFKloAJAaMQblMw4iIQPNpx2NyYyIuZQg4gmteZWOSImJONIQsxsooQUOah6DywzPmENBq3k59QEx4oCg9CZqbHqrhdIqCShIkf673H2P7QMU4/78E1VFIyxrHHRKuY4AXMDk4uPq0UaWIZh5quGxgHT5YlbLcVWZqwWi2o+pZ9W5NnBUVRUM5mB5ZukmSSO6fiOKSNyG+0HMPEGgyKse8jo15M3XwkOvaD5KpV2x2KwDAOuLE/hGj2XY9rFqTWMIwjv//dbzm/OGd1tBai5NBjjGgQ66rGjSN9XFiAeM6PIcgY4QaMtrjYNXsFXoOxGpUmmDSC5tbInVYppkDHB2Jr+EQyNdI2e1QQq4126EAnBJ0STI5OMpQSp9PZbMZmu2N7v8OalCwr+HB1y+t3N3z1/RUfNhXleoExMJ+nvHr5lGKx5jd/+B6lDXd39yxnM7766o+cnD9it6vIZzN6F9Aehr5hHDoeHS/5+atn/OGPv+f585fc3N7y+OljTmYlSVZye7+j6Wuabk+iBk4WBc8en7OcF5ycHFHtt1S7PWmWorXhj3/8in7oyfOC+WzGZrPlzZvvWazW/P3Pfs56vWJzv+Pf/pt/zRdffMk4jrx/94b/8Mt/5vzihEfnJyzitu7+5pa3r99wfHREWcxIbcJmu+Xu/h6lNc9fvOD8/FyCVq1FAbc3N1zf3NC2rWxH00SuFxXo+459tcdthaS7WC4oZjOSNGMYRra7PZv7DW07kOcz2f7l5YEHqJQizYRW8SMXqZghF6Y03SAuAEpjtSZRMsYlSmG1lijxiC2ZIPkyOkihEiRLUCfBpSZXkijmDUrGtkAc0uLPi0VAf4I3qU/GRfOJNQtqKk5/+kbCn/zln2O9xpEzssBl3JStx1Qkg5c1q00SbD/gjftBGqAPE0UD2VYp6QpG7x9+pFJRfOzR/SDpxt5Tdx3buqKc5RRlQVFK+CZKk6fZwdxvstjNrbCKvReu1O7+ls3ttYzlWYpJErq+Z7O95/LykrqpuLq+YlYWHK+WpInw31JjePPNB0mmtQlZXnByds5suaLtWt6/f0/btxwdnYgnuHNopNX3zjH4EabPYvL5Mko0bV3/SdF5SAvyKFT0ZTdWCosOARccioe0G6MVwY8EN+CHjqGtCUFwudEHEpvj9YhJS9L5EX6UbWdTt9zdbmQ0VVA3Hdtm4PWHGwaVkBQly9WKrt5wenLCarWiGzxv3nxPOZvR7PZ888evWC3XvHnzjjRNub654fjkjJvbO6yRm/c//MPfk+cpxydHLJczZvOCx48fUcxnvL+8ROvAh8s3zGcJp1nJq8cXaOVIDFS7DSF4Tk5PyTIh6SZJwvHxEc47xkFcD37++AnFrKRqOmbljL/6+d+IWWHf8d2333J9fcPQ1XR1wW6bYhQM3cgf//gV5WzG6ckpbdPy3Tffcb/ZUMxKzh9dkKcZfd+zXM7JspSry0u2ux0hiMuE96L7e/rsGbOoz6z2e+qmIc+FWa6VxY0QgmYxW3G8PiPNxJEzywpMmmCVwQdH34vV8G63+Ytqz19epJQ6jE7TuKODoBw6kvQMXqQrELdqBo9DB4X2ikM2cZAIdPEpF+M25WOHFTsXzRTFLo4KRHxJLsypOD30Tfogg2GCxf/Mu/jTbumH7+/w+0/+PBFAlRLXRptYdMy379uWrm0lkZfIydIytkw+W8LxGfE+ujNE8DoocLHD8z7g246mf6AyoDxV17EaPU3b0/eOoXe4XixkCErW6KOwy7MsF4cAbWjbHe/evuX2+pb7/ZbeeWya0I8D3333HTc31yTRZ3roWs7Pz8izFHxg6Af2ldjEdv3A+uiIX/yrf8PJybHwqKqKqmo4OTmhzMtDkSKSYREyhwDhSnzZlRbbGQif0Ao04+DkuBrRMk5j3eHziFs9pWUz5PoR3/e4viV48WLXRsTTaZKR5nPK1SmkMzwj3im6tsXY9ECa3dUD/+3/939C5QuC8QTVijldkUuhHUb6VugmfdPSFZIMHUZompbMpriuJ7VRQlTk/NVPv+SnX37B+7evefTo4uAz/vnnr/j48SOb21t+94evWJcZT05POJ+XjE3FMPRcX99xcnrKOI5s+1FY69ZSlqWkA7mRNMs5XR/Rj45us+c/+/f/GcVswXazYRxHPl5d8fq773j/7j1t25BnKct5yetvv+PN9285Pj7mxYvntE3N648f6IeO+WLGfLnCGMPx8THz+Yz9fsf79+8I3h+2fGmaUsxKvIJ+HNhfXkag3DGbzzk9PWU2K8UaOC3I8gKbiMGiNalwq4KnayqqcaBta27vrrm8+sBmc8d/8X//f/z/rT1/cZHSTOTKWDBi52QiB0YFJ90QsSsKcdMWN1pRkfZQBUL8jZaLX0I7J58Z6bqkCEoRCocipT7pnj7xOD8UEvWnL/2AFymkkTpgYdMGaRrbPunCdMyvn9KZQ+yANLImlztIR1PXNFbGLLH7lbnUBtmIutHjDi9CujOPsCp6N+D6VrywjIyFn2J83Tgwes98PmNwjrbtcN0g2XTeUcxKymJOYqxcVG2HdwNdVYnMSAW2Nze8fvuO+/2Wuu0ghoduty37/Z5hdHz93YdYDDjkFM5mJQEPbz7y+v0lf/03f8P52QmnZ6ccrdb4YeT09CwSeXWMaY/HSYVIwvUH3poPkzdYHIy1xnmJ3dLRTXQ6TA9d8hT2qmkqGav80KHxWKNpgxjSuaBJywXr0xXtELjf1QxeLG9Gr+mHEZOknF084Z9/89+x2TWcr855d/2eqt6zWhRk0fsrSRPm8xUnxxIVZZSnqmu8EUH37c0NZZ6RWsNqMePZ8+c8fnQR7XrXDGPPcrE4HI+mqXh8esKH716zOjkhSyw3t1ckRjG6ARcc2902Qh2Kz1++ZLvb0jYt+WxBO3i6ZuTy7nvSLOO/+C//S6qm4367wxjDr375KzZ3d3RNSwiBV68+pyhSvvrqKxKj+enPfsZiPufd+3dcffxInuV89tlLur7Hpgnn5xecnpzw4cMH/uXXv2a321OWOWVZkOclRyfHLJZLbu/v2NxvCUHkUGfnZ3z+xRecn55hjRFLmtsr8iw/uLu60QvmGXzMCxjpO0lWHvoOYx5uSv9bX395JwVMVGX1J38/dRFayYXoQdwBkGKlghaSpPKxUGgmkycfC1QIAaJvuQrRiD54ppDxiVJwKEpagGCDdB6KP1OewievfdoYIgVAEy8mJQ6O1hjGUR+sfT+V0MjPjHf0w/tWlEVGW+R0RY4bB2GAe482sqGa9pjBjSTaYEJgUC7a0sgW0cWRRTkXSZA6FubAGBSeRlr+okQrS9V1vL+6YlfvWC6XLBcdy9ma5WyO1ort3R1Xl5d88/XXfPj4kf1uTzf0WK1ZrNZUbc/Ndi9dmVMElR7yE5WWogxQ1x0Ejw6BbfWG775/y5PHF3z+2Ut+8bd/x9FqRVd3PHr0mLAMLGYzAcedw5iADwPKS6DqhFONwrEg6ECuDUVeitgZBDBHAKtAiL/K6/GjjDzELXLwktXo/AAkzMpHpPmcEKBu92y2DauTRyiT0vuW2/t7nj59zm9/+xW//vUfOFqe8OH9R7q25uLsmMRqEcOOntu7e/KiQyk4Pz9lllju7m7pvOX9h2/J0oyj5QmL+Zz5bCHayKLAuZH5fIa1C9ZHawKC283LGdubW05OVqTacnV5Rdf3CP1rpGk7fJDp4Wi1ZF/t6buB9dEZ19uKdx/vGH3gr//mb/i//l/+c75/8xq9r/Cu5ze//CXfv37NerVmPp/z85//HDcO/ObXv+ToaM3Fo0eE4Pj2m68YnePZ82dilphldF1LWZZ45/jjH37H/f09Z+enPHnymNVyjU1SHIHBOe62O5arE376swuOVmuJje86bm5u+P671wfnhfl8znp9LNpMYw/5esaaCKrLdbyYz5iVsz93xf7Zr79cFsPUsk8dSZi29PGqndwtRVM2BSfYCFRHjgEHrndQeD25BIgezUXTKBMigS82X3Lvfeh2pvNZupdPIfI/86oPr3H6ueEHxWtyTTDWYp3Be6E2jHF0mUZLM/Gr4s/zWuGzTDCjthTFuVIH/+aDt5VFnB9iaGfTtVRVK9HxwZNqEwMu/SECaELhnOZgbTz0jiwv+HB1Q9e1LOc5jx9d0B2PBCdbFQ28/3DJd99+x7ffvub+fotznqzICMpwt6nYdT3NKAEHROJjgCjgfYjKmjaedfMrnAABAABJREFUQQVh/IfAx5tb6qrl44drPnvxgp98/oqyLMUGWsuNx40DWnlGLeoAhfhnuSRl7IXQihE+WVbM0DZhjPYd3gtZ1ntPasXuRYIt3KEz826gbSqKVDOOjjTLCWOgH0aapqZqGrKsJM8KulHSsgcXaLqBX/3mtzG6SdG3HbM8h+B5/OiCIs/56g+/Y7+Y8+L5C16+eM7m/o5Hz58x9D2Xbz7Qdh2L+ZqL81OyvMQBf/t3f0dijYTDup6zszNGN7CJIaLWSgDq8fGa3/76t9LdOc/YjnQxZsxaEWhXdYcKO4bR8euv/onWG54+e8F/9V/937DG8Pq711ir+cd//B+o9hvub29ZzMUd8+z0jLIsef3ddzx//oL10ZKPHz/gvefk6IhXT5+K51TwonXViuvrGzb3G0Bx8egpibFkWUrbDsyXx1w8fkySibPqfrPjw9uP/Pqf/4W6rgRgV2JXXdU1fd+zWq3Et2q+ZDabMV8uWSzmkTqSMGlqtbH4MdA2PzLjfJw27SGgvJCe1XTRIzR7lBcgNISYOgxB+UPXIp2M3CGnkHSFindGudvKRQlBidDGRozmIbrqQTt4mCD/LEIuX1KfItM8uEOHJP8WN5ZMVIOJiyPCZ9nChgNYLts0ec1OKQZjKdKcxWwm45ZN6BJL3Qqp0mgO7HgVn3c1L7nT9+zrlraTMIugNFiJYHdunNAbgotbt36AouTDx2vapiXJBE/yWNbHcsE0dUvbNFzd3PP+4zWbXYtOxZGxd577quK+ahhU5O6ridIaP8c4yf/JkZNbjZI7fd2NtM2G+/sdb9+8Y7urmM0WPDq/IE0S8jzFDT3gGBWEJCE1FoLB2JwsV4x1hQShitDZjQ4XfLRWlngzQsDhcIPwcMYg7hTaSzCH0XLn0saijWyKurbCebA2I58vJZi0a7m92/DlFz/nN7/+LW/evGHoPccXx+yHFpUmJHnCyekp15eXGCu2MJ+9fMlvfvs7fvLqFdXmnvv7e/q2R3nFarbgy89/wr5pWR0d01Y1j169oGsrVJRWfXj/nvOLc7LU8vHdBjcM3F1fMwwdaSpOpj6VlKDReYbRgtdoU7Jrem7utuTLU/79//nf8Z/+m3/NN3/4F9Ik4ebjO37zL79mNp+xXq1IreCIZVkCgaaqyZKE1XLBx4/vyZKERRw9r64v2W53zOcLxtHz8eMlbSN2xrP5gjRf8OL5C2ZliUdxd3/P//xPv+Ttmzds7u6w2pBa8USflQXeBdEm+harDeVqzdn5GRcXFyxmSxaLBUVZinYTDpjyNDXVXY0KP7J2r3Xi30zkL6XqgYj5w9OaCBYTN1kORbQ6+WTk0tE5QXnR/U1RVRoxbTPKY4MT25EQDhl3kqv3Ccb0v3qlYYLNYg2LRfRTITFB8LXD98OjtdEYb1CJ4EfO+bg0mPAqGf3QhjRJyLKUcsxRKjCkCV1msVbTj4MY1UWv7xA81hqWyzVlmnB5dUNtB7rRSeepRJc3emHveg/OR1cJBA/cVw1ZmnJ6es4v/tUvePH8KU/OH9FWDe9ev6FrWvreiXWGSQhaHD73lWju0BqjiCOlbF+n0NVJ0O0ni2RkqzatRqYPNShJCKqant/89ve0Tc2/+sXf4YcvOT8/ocgz8J7BO5QLqJRozpfjPeSl6O1slNJ47xmdx9pAUO6wTBjcwDD2JDah6xu6piK3iq7vKJOErmvQNmG2WOCHkX6/l/WNSVgs1gwe3r97x8npGdt9zW9+93vSrGD0LcpIAs/pxQUY2G5FmJymCSZJub69E8LrIJSNvusZuo7UaL74/HPyLMcRODpagYLZrMRaaNuazW6HsprzRxds7jdcX32kqfeMQ8erVy+5vL4jeM3oRrrBM3pNFQmSxQBpOuOv/v5LfvLzvyHJSn73m1+SjBV/+N33tH3P3/71z9lVNZv7O5qmochyjo6OUArubq8F7G8ref6+4X5zL0zx2ZzlYoW1KbvtjjTJOTl+xPnFY54/f4k2htevv+f//d/891xf30IIZIXFGM98ZrFx0aWVAz9GFYBBp5bZYs7R0RFHxycsF0vyLIPgqevqEy5hIHixOJpi2Jum5t/9mEWqj9uq6eSO8jEZE8LU4TxcyIfxCsGk0B4VJSIHjV4IBDdiotJ/jFyqEYdRngRHUGDjxSHUA4thImdO0VgK8AcrFAnsBKUmBwH5DrG6TZvKBzJn3BROnVpc7YvDiRAUpy5wKoRKS/aczEKCkQxGkxjQeIbByigpQI8wz90oCSdZxvnREfu6YfSBdpCYJZ1YlJaghLppRF/nR2blgtE5xjTl6PiYX/ziF/zNX/+VtPpa49yecRypq5r3797x8fIjg4diOaNqajo3UJY5x2UhxyJE8z8fHqxglLD0+6Fn6Ad65xi8dNAuaDzTZ6pwSob2XVXz+99/RbPfMfY1f/Pzn3F+foa1luBG3Cjvpcgip8tY8kQ4UcVsLjKMqJl00XFTIVjd0PcEPEEHxq4F7xj6IE4OZc7YtqxPLmjrhroeCDojGCiKEpvmfP/6LWmWc/7oEf/1f/3/5Nmrz9j+5jckKmO+WvDu+or5vCTLc+5ubxkIpEnOfl/z+vX3vHj+DKPg+sMVbdOymKWUZcLpyQqtNadnZyxW4tFk0wQXEnwb2FV7zs7OSdKcDx9+T5ZmnL9Y01RrkiyjXKz44zfveHezZXTRejFAuZqTzJa8evUZf/uLv+O777/D7a94vsz51T/+inKxpFyvuLm85na7ZXSO9XJJmlg293eS/5hYFqsZSZrw8eqK0cPT55+J8VzXs9s1FIXi4uIpp6dn7Hc1797+/2j7ryfLsiy9E/vtvY++0nV4iIwUlVnVotDoBhozAIaDATkPfCT5r5I08oE0mpGG7sag0apkisjMyBCu3a88cgs+rH2vR5UBZMGYdKuoiIxwccU5a6/1rU+85T/+9d+wXK0l0FMbxuMp46pEaUvXb/DBkaQJbdMx9C3BC0lWhxSFQSupC27oaZutvHfBE0KcjryYQfZ9x3opkqCua8Ua6McsUlZpgvIS4x1kc+S0kUUNSjykVbQFjl+zv/njfwUVItfKiSYvOijgxeLF+1jYdi6aCA4RlPDEd17lxG3aHtz20bvce7yKGrkdGBXCftxj9/9KoYzGOIPXHm8Mweu9F7SSSoeOgKZsJ+PzifgWSmHSlCqRUZHg6ZUUKJGZiPvg3hfdxsBRK46ko1FBUeTYiO3VTUvnXPRGz5nPJvJclSLNMuYHB8xmBxwcHvLJxy85mB/gnKNeb+g78R9/8+YNb96+oel7smrEZitJskdHc+bTkXSp1sa2W4bmXQ3XJkGZhL5vcL2lswPbumPTdgzWi/dR1DSitCTOeEdrHW/ev6f4hSY1iiQxlHlOnqWQpnuPeZ8koLR0awF6a1GJZbBWkkhsRxpdHcSHq0FrWV8bJXbBbVNTjMYMAfJqHIW3XaQypIwnFdPZnOV6Q98PPDk/593FBRfXVzx/8YJN1/DZZ5+zXK1oO/EBT0zCw/09o6pks1pRNw1nJ8eAZ7PecvH+PUPX88lPXjKaVqRp4OnzZ2y7XrarVYXzlqKqsN6TLhecnj1hs20wxvDi+XNuLt8wmYzoBkc3DLx++566M2gt20OlQRnNZ5//lJ9+/hnfvvqKZ8+Oub9b8e3X34IKzOaH3Cw2vLu6pW46jk8O2dZbfJGLYWKWMRmP2W42BBt4+fJzTCJxYWni+dO//BMSI8TfxeKBV69e8cMPP7BeLXFWHGczY6SjNKD9gMKTmgydCKyT5RkKT56l6ERTVAVZkeOcZb1eYuLiB20eb73AfoLpu462baKFSy+46I9apEKI2jeF1oIRpSpQ7LqOIKZwOgLAQhmIWjwlGzvhvogOi1h41M6P08POhSAEFcmPQuX0gNNRN+gsqEAS4tfq+MXIWn4Hqsvd5D8oUvsShdxnjyZ5Rmu8frR7cXEM1FoBQg0wsZMS0XN4xMWUpihynKuAeMFp0Tsp2Ic7hBg4OvQ7prpnPB2J/gnNdDqi7jqatqNpxRLj9MkTjk6O0Uoznc95ev6U45MT8iyDoBj6nrVdsl5vWG02XK+WbIcenSbUzQYVAmdnx4yrAnxHliSSu6cVJslIioqgC5K8IM0rUJq2bfDOUdc1y+WCfLWirrfUbUtnxY6ECO57peg9YD3fv3kvPKOu50/+6I9ID2aE6BrpvMdHQfLOLM8OgyxZnLhKSkiGSFkGO0h3RcAaBypghz6+H7BYrqjKgtVqTZEXEoSRZjRtTdMNrNZbymqM1obvvv2Ojz/+hFffvpIxRClevXrFn/78zzg5PsY7z2effsL19TWz6QxrB8qyom1bri4vMErz7/7Hf0vnNlzfXaLNE8pRAWnKqBqxqbfkeUZZjlht1jx/8RHKGDbLJbP5DN/WvPzoI25urnl/ccEPb66o2x5LgfYyKRzOJvwP//bfMp2M+e2v/4mq0Pw//2//FxLt0M7x7Pw5d6stv/nme5rOcng0xyQJRTWmHwaycozHc3lzz2Qy5aOPP2E6mTOdzTg7OWG73fLVV7/h+uaK2XTMcrFEK3j+/CnX14bF3T1JmUuOYpKSpyllIUTi+XyGC5672zsWDwuSRHN6eiL6SCPvbz+IS4izFjtY0kJHjDGIgWXgAxdOFYMtJKHnRy1SjRW/g52sxBmP9gkZUBITUXaKf2Kh2PdQcYUfwEdHgxB2eFH0TNIK44nEULff4hm0YFpRexfije9j0rHRBozY4Dpk0jRm5yu1G20irhPnYh+ij/mu1CMM6aAN5DkKUe57FEr5WKR0LNDyfZ1TkrLsFUF7uQH7XMpYEF+psszxUcgbYghC10lMt8TRd4zKirpp8c5xMJtwfv4U62VDOhqPefb0GU/OzqnGY6bTKXkuws626WialsX9ktvrW95fXXN5d8um79B2oMwTzo8OmU0LXC/8qTyXRYRW4rJZTCaE7ABTTVBpQTkWkNVozXq95v7+nu1qwWb1wPXVOx6Wdwxdg+uFQuFJCEphg2fbWX548x4TFIfTGZMiJ9EaH8dXo40Uq+jgEJzC+p6u6zjIZvE1kbCHvm9JsxRre1SWxsQb0Ttu61rY7t5RFGLUF5CYqq53aDOgTUpe5Vzf3PDkyTlJlvKLX/yCZ0+f8dWXv2U8HvPyo4+4vr7EWcc333zD0HUYk3B2esbBfE6iA0PX8T//z/+e77//hvvlBS8+eUZaGqzvmc3mWGsZxQhz6xyz6ZyqKrh/uMegKMuSxnaEwfLD6+8xackwOKpqwqoW37BPX77gL//iz1CIM+3N+x9YPlxyfnbIZFyAVyw3PRe3a24WW6x1HJ6eMJ0fsFgsydKMy9s7gvf8+//p3/PF518wm8+4urzgzevv+fWv/oHr6yu8H3hydgphYDxKsb3j4e6BvmmZjCuU0ljboZQjHWVMphVKKezQ07QtWZLy/PkzslRcK6z1+MHR20GyHXeCNFWTBy95fUmGipI1qxTeJYxGFUUhKchJ+oeVnz/c49wKEi+4jSS9pj7QaU3nPakWGcQjavNB16LUvrjpuG5XkV+wk9sIJrQrgo/JxErJ2BVZM/E7B/BikxK0JzgiByNuFJ3ak5eF1R4eH1XY/fJ7ITPhAyoCiO1tAKccwWv2xi0qEh7VzvZDip4bPMYkFEUhI58SjyfvA6PRSIBw50iGgRACjRvIUum0etsxn0/EbibLmR0ccXL2FJPkeBTjyYT5gRDqtNEE5/fEz9vbO95fvOf9xSWXl5es1muGECgSw3Q+I88T+u0aXE+RpeRaeEtpmlCUhqzMUaM5ZnxCMZ0zmR9K/p9JmA8DJ21L22zZLO+Zvf6Gb7/+BQ9Xb+iDAxsYvCKE6KwQPJ0N3C2WvP7he56cHpLmGSpJMF5iv6wTOdRgrRwcSryo+k5W0eLWKHhGXdcYDZ0TG5yqqvYFvhiPWCzuZcTuZPVd1w1lWdF0naQRW4cxCZPJmG+//z76KE1xzvPZ55/ivSPLMv76b/+KNJElSNc2ZNkxidH0bc1/96/+BZcXr1ktb/joo+ccHZ0wmx1QVhXz+Zy67cjyTJj/g+X46Jj1ZknfdWijaJoapTT/9KtfMj844od3V2R5SZZ5dN/z4ukTnj8/JwRHkeX8P/7v/1fGieJnn31Gb2vs4NBJwXK75eZhgXWejz57yYtPPmK7WZJWOZPxhJ9+/gXPnj6la1t++9WX3Fxf0rcSMzWuSrIkMJvNOT6a0TZbHh7uWC3X0rETaLctzlsm8xFVkZEYxWrxwNAPVNUEpTRJloEH21vQVmy9o++5dY7BOzrryH2gtwN934tHu5JkzBDpNYlJSFLBf5PkRy5S8Y6XiykovBPwe/COzgXSdOeQqcRqI24CzW50Co/fQ8WCIkzzWCDkmt1r+XYJwQLK7xqz3bZrV+QeSX9OefQHiTPe774osCM3gFgH7/Ev72L4aOzMws6mRR6HUkI83Fm46Ej01IjgWacK7y0+NeKpZAyJUbStPKbBCr2gKEe0bU2wIvRtmoaqLJhMxtRNjRsGjp4cM57NmcwOODo9o6ym9N5LhFBRynochQtiYN90LZvNmvWmph56VvVW+EaJYVyVlHkqtih2oEoNkypjNMopy5QkyxhURlbkVIeHhPIYVY7xaQV5gc5ySqXJvGcSPJN6y/jglKqc8P2v/5br99+xqWuw4KyoDbwODCg2dcu3r77lkxfPGI0n6CQheOHUhN2u1Tkh7GqDSuS9ksTgHtt3GKMIztI0bUx/Cbg0Q6PwKrDZrGlb8T8KRnNzd081ipyrbU2eJ9RNQ902fP3lV0Dg2bNnok+bTDg9OmY2GfPdq1ccHx3xp3/6J3zz9VeURUlRlaSpIQwiju+aJc+fnolQtjpgOjlmPj+id5aiyEnyTKxM4vuzfFhQ5QVNvSbPUxZ3CyaTKdvNlsVyTZKM6PolT87Omc/nnJ2d8f7Nt/zy6j1//MWnGD/w6rvvsDienD+lbzoub+5QSvHP/uyPefHxS7SBP//zn3N+9oRmu+Vv/9Pf8vd/97eMqwo3WPquZbV6oMxztHPMpmO00ly8e4cbpGOdHxww3D7QDw1eaY6OTynKHO8dy4ctXdeTZAnLdU0/DCRJxng6oRwV5HlBWRYUWcqkLAVrjONTYsTa21kvSTwEbBBNZ5IkAi3E8TwvfmSBsQEIO8qBYEuDd1il6FWg85Knl2B2HnN7+cqHhgw70FZrDcHE8Q608ZhAtBj2GOVk/PMW5QXA3jVESkkxUUF8iHxcpe9EqSHyaHYUhEer4Ph3ke2968z2uIlzOO/i6j/En6X2DPQdV8so8TXSQJYVoDJymzNkDe1Wupw0TWPstifPTXS+lOSMtpWOYFRVnJ2e4ayF4JlOphwcHDGZTEmLEakHnSZRCyUGOLIS71gv17RNx3qz5u7hgbppCHiqsmIyGskSA09VpIxyQ56J//qoKiFJCE6M9HtnaeuarnHYxTbKfcro+ilaSoUi6BHPnv2UCfBKw+u337HZ1ngC1okflg/S3d7e33N9dc1HLz9msJYCIek+wqQ7DyoVN7PyOjvnaNuWvMjwzkqQZ/TQ2mGbIN7xfd9TpSNWywVZkslWcpDk4p1z6i9+8Qu26w1pmjKajOj7gclkgjGGtmm5v7/jZz/9gtlkTKIVZZ7RtS3eiyZvu90wn41Zrdacnj/n2bOXZOUcMJSFOGrqxJAYQ98NEevRZFmCHRpsb2nalsl0yt/93T8wnh7z97/8mqQ65OnTp5yenvJXf/UfsM2KP/ujz7mLXKqQlrx88UIOOt/wR3/6p5ydPeHTzz7l9MkZm82Ku7s7/s//p/8jq4cHNpsNH734iKHtuLy4YOh7nj59ynQ2odlsaFqLSRxNa9ls1mitabsFdw8LkjTn6OyUYjwmzwvSNOU4zSnKgnZoUQpxD43ebV3b0HUtCk/dbklUIm6hysSRHtQQREeaZsKP1Iq8zMlyCcRIk3TfvPyoRSrDCG9mt+FBY5GA0C5A6jXGa/IP/JxM5EiEuMHaIVSPnCMiyetDzZoX5nMcD1X0nAnBouK6P8T1+Z4T5WPCsRJAN+xY0+p3C9SHvKr973uCaHx0XorYDnv33u8jsiB2YkoKpDFGQiCMwaeeKivIswJtjKxhvUcRyNIUow3bpCZLM4qiwDvLdrtlPp8xqipxBbA2SqDlpk5SKXhKoC95TC5gB8fiYcVm21LXHQ8PSwYrIGyRp1RFQmqcBGFkoLVF65QQFF0PeMW6G2jre/qFYz28ZdkEei8LhSIvGE/GDL2A2FppqixnngTSfs38YMRyVRCGhr4XcHSIQa2Dh8EG7u7uJc2kKOOKQbDH4CUMYgekhiCguMLTtw3BDdzfLFivVhij2DjHbD6TdbUSztVisSDPC5q6ZegkKuvhYSU20ntMbcXNzTUvP3opnKGu4+bmFu8879++E4uRUUXfdXz51ZcURRGVBRq8ZzSuIAWfBAia0xPppnSek+UFWVHu/ebBkJiAGyzT8YS6XlNmOffrFeNqxN1dzV/+y7/kr/7mHzg5OqWanXJ7c8Pf/d3fcTCtOD465mGxoMhSTp6/JK9KhsEyn8351//DTzk+PiZNUy7fveOv/1//geubSzabFScHh7QmIVWG6/eSHBycHHZ9N3B5cQ0E0jTlfrVGKZhNZ4zHI8bTCf9iPmU8npAmsrhwFrq+Y7MSl4yh70gTzd1mw3qxxPbiB0YI0b47oSyl+zRZIvy2XgJ18yxnXE3IyoLxdAJaOI9ZjJxDm5gg9CMWqTLJCEhum8PFSCWH8xqrA30IJF6hvBZfqdj2KIGPRM+3Lw+PyR8qRKuW4CWMYYdDxXFLSsdjoWG/79u5eO6w751H+c4B88PitBMwPxZQ+Qe9tzsRW10VSY1CbLROGM7A3gFUTnxxxCRLJexSR4Gt89EuBpqmFiuTSE50Trx3irJEG/m71BiWiwWjasR4NpMI9UF800kdSZaILXL0jQpIt7jdbrm9u2e5Wu6/xnlPkhuyTFNkikyB8UJI3SW4OBS9VTivuL5b8n5xwWYwNFbR9B7vYrBCmu7fn34Q3/BRlXMwThmZjjLpOZhV2K5lvaxxATor0qbBeZKiYN003C3umJ8coUKQDWoA72XL5xCem5KtCf3QkeWG29sHVsslQ9eTFwXbpqYYlaSDpDAP20FGTS3mfzpJeFg84ENADZIe5K3j4uKC09NTDo+OJG14sWQYBvpuwGhD07b85CefsVwuMFoCTc9Oz/ju++8Zj0fM51PK9IBvv/oN4/EEkyRU4zGmnBIQd1iTaMEenYSipmlKUaQ09ZK62aIVLBYLJuMZ21Dz/NkLfnh/wzdff0Xr4Mmzc+bjCuVaymokRVwlHJ6c8/nnP+Hw8JD7uzv+4Z9+SbPZgHdcvH9H19VMJ2Mu311weXGB1opnT58xn85QxtC0HZtNzeHhgTiK5ll0OhgztB3Ndk1db2gWK9qHBdvtlq5tSUzCcrmg68QqJkkMm7aRw6odIu8vSsNCwNuBrkUO7U6Dk/ukjAkzo2pCmhdojGQvxkAT73Rk8fzInVSp5SaR4iSsbRt09I8yKC3au8GJNGaXRbcrKbvy8uiKuTtd/Z5Ls2Oh7+qSECv9/htIdp1s63YdGhEI3zHaf8eORe1+XkDpmPmmNB8wt+T3XcGMo50PGoKNbZV8fxdHROmqhNplrWXoRbybaSMx6SSU1UgA4b7FxwAGFzwe2V4Nw7DfkmitaNsGB4znR/HGM7EzlLiqNBEfeOdFZLvdbthuNvTtwDBY2r7HRzD/cDoiS8F8aBqIYbCe3g9kZcW6tlzfr7m6e2A7yCZxhyloNPYDgbVC0SvotoFurTmoDAfjnMloxGQyoq07BjxpCHQ2stSNkDSHXh6vdJ8ysjshZZEYdhrzOHrDYrXg5uaKqijpg6Opa8aTsfhh2QECtG0n0fSJ4e7hgcXDgvFIOFN5nqJDxD6ShOOjE64vL7m6uuL07IyvvvqKru3lejFSmNpW/JiEx9NSFhlPnz6l3a4oiopyPGY8mTI/PBIszzkG5xlNxGo3ODnERGA7om+39E1D1zRUoxHnT57xcLvk4X7Dmzdvubi6Y1wWTJOMUaKot0tOT46YHR7w4uVLnn/0gs56vn/9mr/9z/+ZerPh6flTzs+fcndzxXQ2RasJ7978gB0GXjx/TpqmspHMUkbTGXk54cmTp2Rpwg+vv+XND9/z1W9+jXNeTBOdRQXHqCzYbtdMRyNQirvVApMYyjzDecvQDZJ1WVYMidBSfKT1EsQLLIuuGYk2lKOS6XQmKU46Aa3oh07i5pCil6WGNDF7+dmPWqRGiVxk3isJp1QO5xPJVotuBCIylu7H+4CLI4oOCq9VjJcK+4tTBanMOop3d9Ys+23e44JQLIujk6f8m/+dIrWj3j8qisO+SOqoGxQe5s5AWF5otdvUhejwqYV0KrFRAW+k/Y+TKTukCwQA3nUaWimSaD1hjEGXJWme0jVbAbOB7XZLlmfU27VY6VpLWeacnpySFjEKXMsYmaWpnGZGko19kCBJZ3uGvhe3xIgbuchLq8qKIk0wOHRsyVOT4oOK3JxA6NesGs/9qmZdtzid7kddvVMK7CgbkVyrFDjv6FDYvKKue2zfk5mE0ajA1i19cAw20A2ObQNN2++z90J8yZRSJGkqa2kvW8a+70nTlLbreVgs2DZb5rMpy0VPNZpIYR96dBAl/brekmcFoR+o25bj0xPs4Ah1AwG22w1GG25ubsnSlPv7e8ajsWQPas3ZkydsN1vG0wlFWbJYrhj6nqOjI9qu4/j4iOViwXr5QFmUHJycSdDFqBJ1gB04ODwmSROGVqgjWSYYWpEX3F2/x2jFyfEJduh5aDuGtiVPUk6PTyirCatNzWAdRWZ4+ewTzs5OmU1m5GXBV1+/oqxKEq35yaefcXg45+H+nsv3F1y8e4cdOs5Oj/lnf/bPIIR9dz4ZTzAomm3L3f2Sv/9P/5nvX3/HzdVbkiRwdnZClqZsVhtUFHA32weSJGW1cTw8PHB0LIekd/L+i3Om2h/GAEmaUuR5tAIeBHcKYrkDWjzN6aXDDIG0yKkYkeYp1qYMg1jvZGn641MQxipuxzTC4yHgNYLVRM+oXZS5R+QUXvk9bI3XMhbtl26PhEilQjTEI9quOIhscxkNAz5YQtQBhlio1AcFatdJBe/2I5/UogidxxtQBeFU/X6rKcVK0l1MIh7hSvko5OUD1rrig4olIZ1ETSHpDtuV8TAVfKRpGlSA8XjE4k5kMk1T03cGHyxVVfDi5cc4ZVivV5AWKJNFG5MQn7dwuVw0ZvM+YNKM3oq/N8Q8wugfr72Mk4lJpHtxHo+m7jqW255129E7Kb5ySOxeh/iH+FTDbgmCwjto24EUI6LnLGE8ruitp7c9nQ5YC3XXs227+Bru2nrxoE9Sie3SiWHwDpNKinLX99R1zXQ65frmJtocG9abmtF8LtSNthP73pOK1WrNZDIlAEUlzO+h62jblu+/+47vv/+eg/mcw0OhVVxfX3FyckJZjfjm1Sv+t3/+z+n7ntF4jB16JtMZ93e3PD844ttX3/D0/Fy80MsxRTlCmZR+GBiPD8ii42ffSmxZ3dSMJyO6ek2eZ0xGOU29xqQpZ8eHZEqz3WzxQ4ftag7mI4qqYnZwyGg6pSwr8rJgfnjIZ59/TtM03N3cUjcNV7+54OLiPXkiST0vXjzl6ZNTFIFvv/mGi+srkiyj3r7i7Xc/kAXNMDiarsP6noNZyeHhlOmkZOh7bGIYlxLqWtc1Xddx/3BHkqSs12vW6zWKnY9XItmSXYdzgbIsKYuCNEvp216uQ+dIkxRQ+821SY24gxQlVXxvJEAkyPUZoYsQfuQE4/GeGK4YHPTRzM0jYYdqd/MGWasFAl4rBkJc7Tu8dnKP6x1m9TieqVhO9nyoGPogAQLxs8KuC9sVJB8z1ZxQHFTYd0co0ctpFYtU2P2EIHdiCPsO4tF9U7hQkjqz82OO40js8h6ZFPJi78Y/5zwhiSNuHAuJcVtpkuLtQJHnMTHDkLiErmvI84SLiwtmh4ccnz6VcEs77MXbkvm3s4gJ+4in2/s7ri9uWC5X7PDH4KRzyZIUnIx7Qz8wuIHew6Chc5rlek3dtMJx2uFW+5Qf9TsFPCD2KbgBpQIuVdgkoFXchjJglCT/JFrhtayjBythkcorjJIYDhXHyJ2e0iRJ1HWJ1YpODCatWC3XZEnGar1mfnzE0A+COaHIsozlYsXp6SmL5SJiJ4FN3TC0LdfX1xHQ9jx99ozVasWrb19xdnaGc5Zf/fo3/Mmf/ilv373j8PAAYyTOKU1TfvrTn3J58Z7xeBKXRBCUZjSaMPSWcjwhNdL9bTZbpuMRhEA1qnBWdG+j0QjXN1R5TrCW1nvKMufnf/pHHB0fEJTGBvHMT7KSohxTTSboLGdbN/z2q69p6i15mrFcLlF4/uIv/oLTo2NUCKxXS7795hseFnesF0uKqmL18MD93QKF4uruBtsPZFnKfD7m6ZNDijKhHxoSE3h6fk6apKxWG7TRjMcTqmrE7e0tdV1TFDlZuotN0XE7J8L0tmtZbZYSIBqB8zzN9vdRmqQEE0gyWSiNRhUHBweURRkF/SKXGSKO+qNjUtW+SEUhsAv0PjDENzIQxI1zv+oX3GqP+4DcOFFdjwJlFDtBaSDsiZLK7WwdPih++2+9K2Jia+KceA35WMykM9hp7uKn72KMd4UoPP7+ux9qf6NqozHBEKxsr3ZWM/Lkwh6/2q1n9yNS/DUMNm4hYwpzkK3WdDLm4c7Q1k5igABrBx7uH0iyirQcM/gtab4BNKlJ96S3HQ7ngyfPMqqqkk7P75wS5HXK0kwEvd5hvaMPImfp7UDdBbZ1w+CspHUEKUIaBTEKbBeAKpIeH69V0UcO1mBtIM+kA+67hhBkpN6Ni+LxLrCA9W5nYC8BG5GMuoshaz4Ymdq2Z1wVWOc5OT7k9Zs3HD05wyQp9XrNaDxmu61Zr9c8f/FiL1/5xde/ZDwakxnNeDLh++++5cWLl0ynU25vb8nzkqPjE96/e8/Z2Sl93/H8+XPquuazzz4lz1LWyxXr9Zo8z3nYbPj8859QliXz+RylDW03MJ4Z6romSR15njLYHqO1hMQ2Dd4KOVJFwflyvSLPUrqm4T/9p/+A856nH73g2YtPmB2doE0O2rDcNCzWNWmW8/HLT6iqgjLPsXagyDNU8Nzf3XJ9eYkKjslkwtHhnKauefvmLd2mocpy3t7eUQ89qMCkKjg6nDOdjLBDw8nhEfP5AdY6+n4gNSlHhxV2kFEvSzOqo4osy/YdVDUas1ytWCyXdG0XN90I4VfrfQhIU9copSjKgiRJ8DiyPGXkx9R1HaVQGSbNxe0C83gr/wEff3CRys1j8bFBHqwJCOMa9kD6I58pJgbvrFYibd55hUcsOSRy+bG4KB/z9bRGY2JRe+yzYn1AIeZqg+2lEwjSkSkt3y9E62JR9uvoDPo4ku48oUTbx6PPVRD8zO2AGG3QWjCfPSUCWdMniYk/Szozia/aDbeGgMcNPd4KD2noW8GSIOJNCRoZF7uuJzjH0HU4rzClwnYDQ9rjtcNnGSY1WDvQdK2YDyrBCoJ/FHQ75+i6jjoMFECeGKwVLMEkBuWEi9RbUd7vKSXO0Q2CYSkjh8vPf/5zfvjhB66ur1E+0kbUI9InzWrUO2ofR83o24X4JC1XG9brDfPjAWstWkn0EbHA97YnyxKCUWy3nr631LqTLEBthMCqE0aTKUmWYa3jzbu3vHz5koflgsRkbLY1wcPTp0/56re/Ybvdslqu+fk/+zneB4qi4vnzKUma08fgy67rKIqCzWbDi+cvePfmLePxmKODA3775W/prWU6nbFYLJgfHLKtG5K8oG1bmt5SVoHclDjnSZKcruv2nBZrB9rNhq5eY1Cslgvu7u54/vyMg4M51XROUY1o64Zm2FCUE2azA45Pn5HkBf3QUW/W0oGnirqu6buOEBSTyZjteknbtKwWHQ93N6xXG8HzHhYsViuKyIZPNSzWK5ytOTs+ItEpVxe3OOv3Jo9du2W72eJcoMhEqmKUITgJzL27veH29pb1ZkuW5pRFSWIUZVVSZCKgt4OljzZEqIDzPcrJ69B3wt3zvozUkAgnBL+/Zn/UIpXuqNxBiotIRESl3juPU/JnF0KMHA94JSEKJoCLmzWjdhs/j7OxKKndGi7s+AR7zor8lRPNX/TOlsjyIY574qCACvFmCo8GdlGt75XeQeX7j11Y53/pI8Cea7UTUe5mKqVk1ZymScwBlKgbrfW+WHq/W69HI+AQxOM5SBdRVRUqiHWFc5btZsMw9HRtQ5Xkkj/nLGnM2nORIR97KfqI30ikekqW5bhBsKqu66ltYHAWm6UUVYE2Cj9YvN0Jv/W+KLvYAaogcpXgbOR/SbSSido4/Idk2qi3VEj3pYhOpPKcvYd+EM2iCJqNLFLsQNCKKqv2ox9Kxxh5AcY3m5qnz1/w7fev5XWKOkKjNcvlkn/xL/4lCqjrhs32juurK376+Rc0TcvDw2Lv/b5erfAhMKoqcUUlYK3FmITZbM719Q0nJyf86le/osgyTk6OGezAmzdv+Iu/+OdcXFzJTejE7yqNNBWtlfDZ7IBOEpwbYmeY0QwtTb3d5w8OfYcdes5OjlDak+YZQYmbg9c5RV7FLfBAMBZHj/MSda6RTXDw0PcDdzfXfPvNl9xeXkAQakzwFoImi9jPWZqgdELfWZb1kiJRjItDttst93cPzA+OyCMWZbfN/vpLkowQO32lFOv1GmetaEqtYzoaUZalEDQzwcaS6Jy63W7FYTZL4oJFttlt29A0BSZN4zZc4AEVI+vCf0Mr9QcXqWIX5KfA7jAMH7Da0DpP7wMuBIbgcE5Ijy6I5MSEIE6NSjqQJI55iQqoGFmklQSESpi2jzl+0iV5JZ7pYpTpCTuPcL/DpYaIq0gpCh6CDqiEPWnMa+FvPRao/0+1fBfCwH6UetQfPnpOZWlOFm+AXY8RvIuR6y42tdILOrezaVGSNJImdI0EbfbDwHq9YjSeYoeOxNm4xevEiC1R+ODQWpEmkasVPIO35HmBSQy+B+cDbT8wLlPGoxFlmuCU3DBJmpA4UEpuVKUsLhDxLrlgdkRd6xz/8T/+R3btgXRRIXKoErIsIc8VudH0bQcugNuN5mr/nJx3oknUhjQv8HGc1DrBBU+W5vvIqqAMWVmBhpubW1Ca0XhK0wpe9PbtW549e854NOLy6oqL9+9lfHr2DOccSWKYHcxZLBYiG+o7jBGM6O3btxL9VYi/1NHREb/+5S95+vSc24cH8uNjjDFcX1/z+eefY0xK8J5uECvou7s7nuQFeqQZ6obJaCxb6SA4jE4Thr4ThrwW+oXtexb393ErJinHpa3I8jF5OcIUE0xekWYj0nKEQ0fYJEEjSUQ6csvu7x+4ub7BqIQnZ2eUuRx+Td2yWKzYbmvqzYa67VD60Vu873uWywXBlpydnWOt5fb2Nl7hcrAm0Y3UGMNgB+qtdG6JMVRFTlVkQtgsSjnYANt3rFcrBmsjobOIoQqBJBiSmKnn3ECzXROsxQ8W2/eYNCXPC7Ks2O3Z/r9+/OGMc/34HbURs7mgYHCQeUPjZPtmvadzLgYMOIL3YlqHtHqplwvd6Ng06ej9qIz4BiGdlUF0eNEzbn9y75CfsEvFdRYbQbgk3ZEeieGc8c0wYlHseQSHA4F9e/UIpe0/pEiFfZHid4pa/OSYtpsm0Tov4mTeO+zQ452EEUixEz8dN/QkiWJwg5w2XUtRFLx//56qGmGSFJ0WHJ+Kif4w9Cgt3ug7Br3W4J2l68QrHaQwD4OjaQdsakiMZjTKqPuWTd/jE2HCK23RRooFbgeYs/eTD49AHvv0neDRWlFVBaOqoCpTigzKNAqe+932MIhMSSEdmxHuGFqjjKjeu77D9R1VURD8I4aojOHo6ATvhpjJtmZdbzGpgOuDtYTg+errr9Fa0/U9n376KdeXl0yePY+4RyxmT59ye3srn/PZZ5w/e8r79xcUVUmWZdGsUHDD+XxGUaQ4J0kmh0cHeO94f3HBdDrhu9ffMwyevMhZLh4oypKuawDFeDwmSTR1U7N8eIDgyVLDcr1C2Q7jHaNxRVHkDNbgkW7RRfKgRmF9wLYtOsnRiUAJWgWyPCdJDEVRMZvO+OInn5IS6OoFq+Udd7e3vPnhDX3f0HZbkkRRFTkugDKaoqhIsYyLjCJPuLq55ubmLoquJ4yrEVrJRi7P5bDYrDfUdS1wRJaSGJkaqrxAac1ms2GzbURQbF2kIxTCVndyTaVpQpqKiBjEtaKpG7b1lqocMR5PBc7R+scvUmZnKxBEzLsDvnUIpBryAF0U7Q7e0XnxA1LIDSA7uCD0Be/YrewkcVdhlHRWCdH9k5gUoyT1OChhg6v47yq2OvK7mOg5vEQy7UIffJDEYBUtYiJQrOLFwo62wI5VsPde2C0AHz8i9h6UkFlNPK2IN5neA/GytTImieBzwDvZlGitweh9IKjRGu8cTdtSjUq22w1N23N8Fjg+e0KSlwQvI2KiNdYOeGcxStF3LXcPd/vvK+C9o+2gLT02WLp+CyiyPMebnJ5AniXkeULSaNna7fzCd6g7xJNhR7sNBB0YlTmzScV4lDMqElLfS7DCYGMxkwIatBB0lTaYLEMlkqfnZCKXOPVo7Nd3PWkqN6U2mm1d46yjbTq224bxuBLJkfc8OTvbhzScn58znUy4ub6WxJzplDdv3tB1nURSjcTb6fjkhMVywWq95unTp7x69Wp/k+ZFQdvWVGXJeDJmtVpQVjnWydi93a5YLO5Yb9b89//63+KGnndv3vDpp5/RDBvmh4eE4FkvV2ybmrapOZiMaFYP2L4ltDXzMic4h9us5aAtJiRpQVKMMIUkGwedCAxilFgqR7JjYhLZrLqBvm2p156H6wtuL9/ghpb1aoEbWs5O5pwcz8U1V8F6ueTh4QGlHCFIOOtm27Ber/HeMZ1Omc/mFGVB8B5rHd6KCR0Qt6UJJomQhha7Z9sPtP0gKdxaiikB2raVjlkpsiIlBOmQnIxThOBitwYuS4nAD94Nfyhu/t+Qu6fjBRtigdBhp+xFB+mWTCxIOyvgwTmU9xK9TsBpRYi+w7t+xqBItSZV8mCSPUwetyTxJtGaaLoXK2XEqJQK+/X8rvNRYhcKxODNGFCJc7F6qz03C4iZerutVMTVdvgLsW/6L6xLXRQkW7ezeZHP2QmMQyRsgopAaIbOFCFVrIqCum6oyorVdsP1zQ1FXoqz5OqB29sbTrMKbRKyTCw5mmbLctWQJGbPP9Ha0A9OopFQtIOjGTw+jnjGaMpijA0JTvWUnWVUpGwyGUtUEj1H93iTvD+7IuWDI8lSppOS6SijSg0pnrbZopyTd8p7BusYnIuLDdnSigA3gcQgDs8BZSRSKaJ3KJ2w3qx4uH9AKQH6r69vJFppNuHm5hofAkeHh9zc3BC85+L9e7Z1zWazIYTA1dUVl5eXFEXGfDbHeUmbHqyl7VuKquTXv/kNFxcXfPTRR7x595bDg0NAfMn7tuH+/oYizzgoDlgsH7i+ueThYcGLFy8YjyohUZ4dEYKjzIv9VtY7h3KO+WRMV695+8P3hH5Lhid00LcdSZqQFRVZmpNmJcqkoAxBmRiiKixBa3ushV7JhnVw4miwWS1o1wt8X2MSTZ4WFPkpZ8fHeO94WC5YrVdsVmsITuxWkpQky+maLSEE8kL0mFU1pqgKxiPRZkrIRLvXUuZFTpZmcYOnGKylj2Z2PkQOoQ9gBAqxg2yB0yzBO8PQS5Zg13UkWlMUmYjxtZCmvbNCcNa/jxL/1z/+G2LW41XsPxh8YoHaUQKIGFEIom0TnozFei++2DpBBRODP8WTM0GRsPMbj4GgkSOkIuFTqFdBrIH3uNCOmRD9Mv3jAwt4aXlkQYh2Cq2la1ByX+31eoRdBp50An4XDOElWebDdJmd9i9JU0yS7HEqAdplJv0Qu9oBhEoLv8dreVxaw8F8ymazpWlakjSjKHNWqxUEKKxnu15HGxmPjaZ5brBxo2Q4PJhRFTk+KEajEaO6oe07etvR9IE+GPLxmDTL6AdHsI5cBcpEU2UpqdFoIxjSjuahP3ieBHnNU6MZT3LGVUZmAtoPeDdQZSlaZWy2vZymNkRwX75fYgyj8ZgsCq6NSUnyDNc1GJ2wrSVUwvoQgXaLs57pdIrRL5hOx6zWS9E8Gln9//DmDU+ePOHq6or7+3u6tuWPfvYzbq6uqeuab775itFohHV2f2CdP31G17V8++oVf/Hnf854PMYOA+vlio8/ek7X1NzeXpMmBvBcX17w9s1rri4u6Puej57/dwQ30Gw3TKZzEqMY+g6lRK852J6yyOjahtfffUdX16Q4BgKYlJCWOK1onZzrSZqh0hydZhCLUWI0xrAfr5XSeNcRrCM1htlkyqxK6bZLFrcdzXaF9471csVmvcbaQXL0ioKD6ZQ0zbi/v2e1WpFmBTpJ44JFSJjr9YauE/qENjKK+0jn2SUXhxCw1mGtj12W2nuk7VKVbJBlkOCMhhBEtuQaj0k00/GIPMvJM5EpGaXwQ09vLVoZmah+1CK1KwDx99ioSDeDF1DEf+BQIHEn+3RjBSQqRrIHsePde0YRfaPCbtSL9Ubtxrp4OrPDpOJoonZ9jhTJELshpcSxwVuH15FVrcSxkxDHRx+dPqNJPF5GQh/HVL+zc4kuCPuCFU8ABfsgUVmvKgxaGO8hEZpEkuBSQ9+Jtst2PbbbkCfi2ijz/JrtpiEvSp48eSo2uQqJcK8bymoirHHnWK029IPY246qEVmast7UVEVBnqWs1isG17OqPXdLw6iUCyVRiizJKBLPKEsYFxmzUYlF0fQfOK5+MOoqLVjeaFwwnRRMy5yR8qi2IdOaIk1iLFeHI0S753iwKJhMxswO5pSjEUob8qKgt5Ysy8VmpZSsQqF/aLKs4PDgiOurS7Is4+bmhtG4YjafU+Q5v/71rynynKurK25vbsgLSUmpqhFFUbBeiwXJaDyKWkHF+fkT7hcLiqrk+fPnfPLpp7z+7jv+3f/47/j2m28YBitZcbbDuYHbm2vW6xXbzYYsSxiPhDfUNDVlWco43dQkSYYm4HqJ79J4NsslaWI4ff6coavp6hpvocwL8U1SMJoeyAGX5zLKmwQb7y9xOrUMcSvqrOC7wVt0GPC2pWsamrqmblrRew4Do/FETPuyVGRaXYfWIjJuuo7j4xPKsqSuN/R9T9MKry3LxGrGdS3WiqWMCoGmaYRh7p3gxHHz6qxs5soyi26oMqITN3fiZuGirCtlNColRceYR/KgkyRrHyx2EEnZj1qk9h/hA17TrlARL07vMM6SekfpAxmQoMlUoFJQKU3OzjNKRMhGIXq83Ri5W5HvKAIKYsxMLEzwgTJViKDxF4iBWmJk3LDe460UKaMghJhCrAQ72clddpYRPp6+u98hav78B3FYWu+1eqlJ9gVLayPjXhyLvfdCwsxyWt1GPlTL0LY47SmynNFI4r1Bc3N9x/npGYcHB4znU3TqWK5vZXNkLcakZFmG0mCt58mTM87Pn7D66hVaI+b4RsiSdd9x82CZpJrUJ8zmE8ajMc5tyXrPuEw5nFUEBffLDYPzKCXDaqIVJk1IsoRyVDAeFYyLlEmWUCq5kbR12FbSadp+YPACAPsdRofixfPnInIdjVCpRM8XZcFgLTrJxKXTBdIspY0H37t3b5lOxrRtTVGWOO8Zj8ZcXl3y05/+jG+++YbXr3/gL/7iL7i8vOKLn37B6++/ZzQasVwt+bM/+2csVyv6rmMYen775Zf85b/6V1xfXXF6coqzAy9fvmTx8IDRUsCLzBBSTd8rbN9TFgWjUcXR0SFpmjGZjEgSgx16gi/p+57JeIR3wv0qq5K721vatub8yRPaekM3WNAp4/mINJeorDRNMFlBkmXo1GCyBJ0VZGp36Kl47moiiU3kQm2Dsx2238Ao5eRoRlsLoVWpiLt6MULcbjaU1YiLiwuyvODZdIpWmvu7O9qmIUkTwaOKkmEYuL254WG5wA0uvm/Q9z15IXQDkFi1JBGbaGNSgUScTBmDdULRsHav46uKijTbbffEYVWrICEn/UBidpYuYgD5h3z8NxUpYSA7gosi3/jiJkqRaZgoERFXWuESg6IgVVrGDBMY5RmJCpSJoiRQKChMIFUOg6zsJXT0g6CD3wO2Hz0Vdl2WdHF7xnckciotaybvHIOVXsFoiaIK2u9FzHtPm1iEnHP7IrUba3e0AwC8tL8qFinppoi8Lgn41HrnhRWpClnKFvDWCXjcDdTDQJlLoUrTlGbd8qtf/pqf/dFP8cpz+vQI7zb0/ZayqiiKMUppmmZD8B1ZlnF4MGcyrhhsoCwKSZ8dGnpn2baB+0XNOKuoysBm3dO1luAcuYFxabA2IbicdngsUmmaiDSizKiqgiIxZBryYAlDjxsGus6yWm9Ztz2NtTQWOisuGB5FlWUcHRwwmczQaUrQYvlSpqJH9M7jrSNP0ojZee4f7ujalulsTJKJ0Z1YGWum0znD4Hjz5h3//J//OXXd8uLFR7x98xbvPTd3t1TjMZPZnMPjE37xj//AYrnko5cfsd1s8FboG35wlJOS5XJJmiaAp2k68J40FfD66uqK8XRC1/Wcnx/FLqqK15enyDPxtgoywrdNzWa94vTkJLqJNswPDmnqOmKyDu0NmYnPJzpcPFoKybVqtImLG8SU0SFmiM6yWS+pNwuCbcBbEm3QOsFGgbt3A33bsKm3tE3LdDZjOp2xXq24u76mqZvoKCsyqMXDA4vVks1W8CplxLCuyLNIJ0jiqCebaufFRdUkkv6zM4hkt2jShiSVww2tcd4xdAMmMyjzqA21g8UPMcLMDj9+J+Wcw0bfohACXkg2aIT3Q1CkwZAGRestg2ylSYBcKTKtSDVk2lCkmlIr+ZUoUuVJcOggQDvexcLjIqgdLYMJ+2KhgyTYiKxuB3THqHfn9oLGHaPaAmG3+tRaeFSxD937nsftkfMS/bQvhkpm7iRJ9v5QwzBgk4Q0TQU8HwaU2dkXhz2rd+hli6G02tsTS1s+kGYF0+mE4+MGlKOJuXk6AZPDs48mdN2a9VrwnRCMsHXjVrPIcw5mM+4fViSJYTIeUzdbad/7wM26ISu2VLMpQ2giOdBjNJQGXKYxkxIbFLtFX5om5EVGmiWkicIER+oh9C3Ndk3fD9SdY9N1kR+n6ZyndQEbaRiTUcXT83Nm87loDQOkWYaNvJqdNAZkUbFarZnPD9hG18jVcsnh0REohdEJfT/wH//6r/n000+Yzw/49ttXfPnllzx//pzlcsFy+SDd1HKJMYa+7/njP/4jfvvll1RlSdd18VBJowI/Y1RVfP/tK85OjiJdpGPo+ziiJMxO50zG48g36sizjCLP8T6w2WwoihzQLO5XTCdjnO3ZbjaMx2MAptNZ3DSLWDfPcyEGK+HSua6V7pNdRuRON6pJtZED1oonfp5nGEYonwmL24kbSd2uub6+5eH+Fmt7CRgtS6wLpInh8OiYUVVxc33D/f3dnjvVdh1ZmnF+PiVJUmycNhRBlgE7bNkHAcatJ81yUVUo4kjYxkCUQJal5HmO0gprHVpDosR5NzEyiiqiM4pz0dHD/fiYlLXChg1ByHpCEhfVvR081ga2vXQITW/pEU1YqqFMlNwIwTBKDIlTpEaTaORGSDRIERZr3kQCBwkJzmnpXILM51iFDuJ7JAHp0eaFR25T23UIQVTvt3PBiyAapANz/tF3at9BuV2aicMPH0ayBwEGsxSd5yRpilZaTPGcI/OyTdz5pMv3FKeFNMtgp4dzDqwTj3Unb1SaaY6Ox7jQSnDC0HN3d0s1ybh49wPHp89ETqM1VTUlURIWURUZ88mIk+MjlqsNRivOz85o6ob1Ggbbs1Ae97DEuZ4XZ8ckiaSsEEfx0ihMkRK0wfmB1BiKIpOtoFLkxtA3LQwDfScYWeMsW+tpQ6ANit5B5xRDkFM1UXB6esrLlx+JY2OSYLJcbkABMAjOyfgTFNutcHOSJCHLMrbbLcfHx5gkZblckKY5dd3wk88/5/DggDdv3rDZbnnx0QvmszmLxQOffPIJl5eX5HnG3/z13/D06TPu7x9Yr1fc392zeFjy+eefs16vCcFz+vKY77/7FkJgtVygVWA8rmjqLUdHR2RFSVmWVGWJUopJVVEVefTED+RpCsFze3NDUY7ouoZUV0xGlRxWUTZltMZ5AaCdb3GhlXlBpwSTYtIck6SYRCx5zE4TF33G+uhVpbUhoCSrzkkP5pFw2tPTUyaTMUrDdDwhTUUXGZxntVzSd/3e2pkIoRRFIdBBYuhikKeLpGhrLanJPjigDUqDd3KYgKdtW+wwkKQJaZpRFCVFkdMPA03biluHtnifYpTBuyBFzbpozaT3k8+PWqR2ntSE6EigwFlH1w00vaMdLIPto/zDoWMoQ6E1pdaUGgoDqfaimFdOGOBe0l5cUAQcXqlIehPLFqUCSaJJTE6aSABlp6DxlmGAfW77jscUxD207TrKqkTFhJUQR0Mxr9vxgXa+SYJDCSvcSrhCnLt3Wy5x5BxwTnCILMtwLjDYjiw1kTcl5CrvrXScXjgoO7/wvu8Y6poE+d46BEyimM7GKAPLh47F/YK7u1vKUYYymqPgSBJomhUqwGQ8xSSKssw4OT0kzXL6fuD7t++wAV5+9DGvX79mvV7Qu55lM2C6DuPh5PSI1Aj9ItGKMk0oTYIycponqaGsChlhfcCgaZ1i0wqPpnOOxnoa66htoB+gtzA4T1CS2Jwmmo8+fs78WCxShuBJlEhJTCqiWecdVZqjY5Hq2pa2E/O546NDJpMJd/cPFHm5t/84PTnhl7/8BYvFgj/+4z/h5OSIH978wGQy4mGxiHQLw2QypyhG1M2aZ8+es9luqaqS+/t7qqLg5cuXbLcbJpNx7AxbRmXBuzdvqOuGFy8q0GYPBs/nc9IkwYcguE4i5m2r1YqyKPHOkho5VOVQkuSYut4w9APdMIDS6EQ85dEpJi8oRlPSxEgRTyS5um1bEfEqBU7cApwbABGpN20nhGArB411YuBn0hStDXk1ivKmjq5pqduOwTvyIqccjei6FtsP9H1H3TYSpGJtZBOJ9CoxhqB9lH9plJJxXO5/wcyqqgJCpDmkAGwj4D5YS5Yk5F7kPsuhw3vLYAdUEN1qkgjZM0nTH7dI7casoIRa0FlLawfawdEP0rqJOBgyI5+XoCm0ojCa0igKA7mR0U9F2YuO4LQG8CFav7jYFg0410Nw0i8F0VDlkwllkbNeJWw3a0lg1R7X2xjXrXBDD404Apio4SPq3/ZJvLGTUrufbXuCE23gY/RV2D9/i4Wevbd1HjdVohGUE1ZOPS0bT/WI4+0sebu2pR4awX9cz2g6YZxV6GlGqnumkynXVxcsFyuenD9lu61JkoJRlZCmEJRDoQlOwhuUsjw5O+Tu4Z7VtuXk6Ii+7+m7lrZ1BA8bXfDuwVOHBQezjMmoINdGRMEGkkRDiKOQ1gTn8M6xbTvWm5rVpmW1bVl3A50NMua5gPNx+aAVmRF6xqgsOD46jHiFjksLzagaMQwWoxPyKoHg4yq8pW5qNpsV3jk+fvkxTdOI22OS0rYt5+fnvPr6a7yD09Mz1psVm+2Ki/fvWSwfOD8/5+zsmPv7O9Ik49e/+g3/6//N/8TX33zJ4mHJH/3sZ5yenrBaPAjPqSwAx+X7t2gcGsdsNpV4+7oRfGs8Zj6bScE2iRAj00RudOuoRhWr5ZLJeCIduBfm12a1ZrVc0PU1SWqEcqAz8ffKCpJE6Bk74blJZePpgoxRvXUfYL/RR0xDnhqm8yPapma7XokDiIWm6VBaMbiWh+UKZ0WFoQhs1itur664v7tDayW4ZVGKPc3Q0fYdSu24hCK2t86h6CnynBAUfifz8pItmSTSJXov0ErXt3KA9QIlJEmKNkJHEOeEAZQoFhIt+3lj1H6Z8KMWKWtt5E6IQVnnhLznrN9LQVBCyjRRJpKgKBJNkSjK+HumApkippmEKIPR4iiwD04ApQK4aHZnrXQ01rKNuWw7E/iyyJjNpiw3a+4fHuidxUd3gKEfsLmlKkphd/vIit5Tnx7/29thP4KF8NhFxVZq36U5jwQHKOmECI5gB/qmpstSqkIU3wrhgqVJgo2YxGg0otvkbNstXd+Rayj9WGZ3pZlMMvquJzFPuV88sHhY8vTFS6qqwroBX29ZrTYkSUZVjRiNSkwiDPblesPrHy5o25qT4yNc3/H+4r2Ys4XAuhvoH1oWG8dknHE4GzEdl+SZBpORJBnWa7YLKRzD0LPdtiw3Nc0ArYfeKawD6wUkByHo7aQ1qVEczCbM5pN4aEl7m6U5m/WGPIL7bpAOoOsarq8vabuGzWbD5z/5jKapcc5JXmE/UJYlf/VXf8XhwQEvP/6YH374nh9+eM1oXHJ1dUmep0ynI0LwlGXJ7d09h0fHfPXVVxweHeCGgfPzc25vbvDO8vbtG85OT1gsFlxcvGc6LpmOn6C15qdffEHf9wSlxAY3TWMXLFdL13UM/cB0OuXu7o7ZbIpGkeaSzHx1ecnD/T1aQZ6LUFfpBJNXjGcHBK1p+4GwrqPvv1gYT6dzCX+dCF1DJ9KVggwKRqlIedEMfctDlrNYPODbgfF0Ltl/UT5Ubzb0fYe1A9Y5pvM5KMV2syYrxMkAPP2QM1HQd+LOIdf6LgdAE7ynaRoG26PNIyUlBE/T9Hs8yjpLQB5jEhcjOkkEFLcDWWbEQ81osjQhiwuRPHa+P2qRartOaPqDYCsuYgv4QBIUShmRnhi938YlSkuRMopCCw6SBk8CiNmocJU0Oy7Uo385weN97KK0bDt2vCVnB9qmxg5CSMuLnMP5jPl8Jorvh3tWDyvRfHU9LYqiKKIvkxRa7z27TBjn7AcFKj76Dxjn7EJLA6gIqmsUfZQBWQXBOYxRuKqiLMu9cHOH3xHjpvx8Dq4nNDAejTg6OMBbJzP7BwDqweEhk9mUtmkJ/oGuG8jykmo0Zug7jFJU4zF5PsIYxcuXT/He8+13b3HOMp/OCD5wc3/HelvTeYfrFe2gaPqOTT0wqrZMRgWZSdAmww6BtmklsCBe5J2HzusIrosrQHSXFWsXJfgJ3pImhuOjA2azKSjFerthOpux3myoqjFlXuLsgDGaerOV0aTr8c7x7NkzTJqyqbcYYyL3acl6vd7n5X3zzdccHByQZQmowLffvuLw8AWSgtNRlROWiyXz2SFPnjxhtbznZz/9GX3XRUdIQ6IN27VE0J+fn3FzdSm2JcNAbx1pXjAaTZhOD2Tr2tb0XU9RFECgGI8Zhp7pdOdxLmv2u9s77u/uJGq873mwNdtmQ16MODguGIaA0hbbe6bjufg6uR4dLLiOvgZnU2y09Fdqd3PLomEX5OFsjzI5J0+ekWT3bNbruKDSghPFDMA0M8xnU5rjE25vr1ku7mWR5D1Ns402wY50PMbsyMeRztu3HdvtliRJGI1GjyJ7FH0nzg7Oyt8ZLelC4twgThrdIOk8RinyIidJc5JERacM2d/L5vAPw6X+4CJVt6LYJ05NPm7UdtYoRilc9BnSkZmtURSJdFOZ8qTBk6NItSJVkEZdnkgrZFwUrwOHc4OMX0GkFrKO3YHdSBFzEoO0WC0EFEwMk8mEZ0+fcXp0yuJhwWazxjk5FaqqEvFjYqKuzu+N84J/BMrlef2uDOZDGoJ3LpobiyUKzmKHFu8d/TanrKrITk6FwzZYurajrhu0MhRlRd3WOOfJ04ysSrHWUrcdSZaRJIa8LKiqau/3M58fUlVjprO55JwFJViEUcySimpUUcXi+O13byUSSivZbkVXgN4L9WLoPb311L3nfrmVgyGAUgk7Dr3DS7BGgN6rvTPqzuE0TVLRbiJExDTNGI9GjMqKMssxWkdfcUOWJRLbhfSu1lrarmO1Wu7TacThISXknvF4gu17BmtZLldMJhPevPlBMgu7VkbmZst8PkcrtcePXn//PcfHR5ydnZLEROlh6HFWc37+hK7tUEh68qprOTs7Y/lwz3K54uXH0rHmWc5sdhDTd6WrGI9HJIk4I+y6DJSiraUDrJua+9t77u8fWNzf0zaNEFwxKNPxw0VNVl1ycDCn2bZMxxParmU+n3Awn3Kfypb44OiIycER5WiCTjJ5zb28XjvTuc1mQ9d15HlGWZWMJjNub665vL5mMhoxrkqUgiwtsCh603J+/pSTY6FFtG2Dt3OatmGzWZMkhjJ6QyllZAPs/D4K3Q6CJ4VIbjZRJyreYTKSV6VwybbbLZvNVoieWst4mGZ7ao71Dt87siTBZEYWZj9mkdpFYRPi4l7tRClESUWI87OJfk5C2CwTRYYjwZMrTYYi1YYkCojFjE46p4AFBvGkcXGjF/x+pBRHgWisthvFkHW8dY6+bVmvVly+v2A2mzOfz3n27CnbzZbLqys2my1plsYXL4mMcxdN5ETzt18Q7LD1IE/wd61OhX9lvTwH7wa6Zou1HV1jaNst3jlGo5kIRUMgNQnBe7mAnSNJEjabNV9/8w0Hsznj8YjpNGMnLOm7ni7pKMqK5WLJMHiePc3puhbdG5ROSIIcCr21eK8YVTkff/RU0mVwvL9oKMsUR4VpNW3bYZ04jbpBNnPSwYorxKO7pmzqvBIXFhuI3DPhvCQx306FANYyLSqUHTjISyZpztBLp+GcIytKRqMxPuKO3gfq7Zbr62vcMNAPEsSZZjmT6ZSmrmWsQbxd8zwlBMd8Pucf//Ef6bqW45MjLi8u+dOf/ykQeHhYMh5L4snR0SFZKvYhX375Kw7/2c85Ozmlqgqc7bG9o+0ajNFcXV+RFwVlVTGbzSmKgqqsSNKUrhcpUp5nELykK0enBeUtDw/31K1IkW7uH/j220veX9yyXEloqw+QZBnd4HEOSDTPnp5xf38Tt1swG084PphxdnzI6fFMtoXjKeODI6rphOlc/NSrqqQsC8E284zbu3uavsUH9v74q9WCtq6pG8GZ+t4S4rbOBwfeYhJDnmcMSDJO3/coMnwieJO1cphv1muAGAHWibFiTH8yiUbrFILwo7ROaIaOZtUy9IJNmygZShPZEg7WopTQiET2FsQV9nf7gP/fi9TuQ7AZdlc3O8mJXORSZQWjEHeEVAWS4MkQu5dMa7FhIY5BEZwWaY3DBydbCy/gubUWb3cYkY2P4DEU4THYU/6sEffJm5trLi8uyPOcs9MnfPbZZzRNy+39PZvVkjzPGFUjyqKg7zuapsZaC8HxO4TRyGFQe/q7rHJ9dPUMIUSCY0vbbjBGYYcegsYNQaK784LxaEICbDLN7bWsY11Uny+XC+p6y2Q24+DggFFZcffwgO0H0mnKwcEhF5fXKKU56npJOM4LXGtJ85S+G4R2ETTTaUVWvCDNDEdHc159/xrvH9BkJEbTDw7bDxKDHXYYoI4Liw89UOMmV56uXKRRFRCsJSgVzeJOKJWhX63JHczLitloRHCO0XhMmhe0XUuRl3jv6dqWLBHr42+//ZYn509BBQ6PD0lNwn17z3R6ws3NDcPQc3d/y+nZKb/6p1/gfRyLreXnP/85aZry5Ve/5ZNPPuPq6orlcsnnP/kpaZLw93/39zx9+oQnT55wdHDAD69fc3d7hcIyGY9Yb1bU9Za+6/nkk08x2jAeTdBGqCVqp7tUmrreYm1PUeQ4O7Be3LNa3FF3lsWm459+84qvX11Sdw7nwSOdQ7AWF+Tg00Fxtdyyqhu6rsEHzfv7Nfr7S56eHXN8UPHsyQnn5+cMaG4WDySXlxwfnXB6fEqSJLRtR9NIRzWdzhiVVdx+JxwdH9Nua9F84rG9bEXr9ZaHxS1ds6XrGxSKqizI8ozTUbU3a+y6nsXDUoTBSSIgft9FLEyuew0EYzAIVqe0oW47NvWGbS2e+UYpEY5HmVTTBLQJGCMLAKFliO1ONvz/Ybv3odh2V7J2mXk7J02jITViJWvwpAQZ7VAkeve5O0b5DvtxUqC8x+PACZHMRdbrvjKGRwGzPKiIH0UZ357BGzszYwz1dsubH37g6uaa6WTG8ckpL549ZblcsomOj1VVYYykusgm45Fkti/28WcK+O7EeVipR7sL7xmGga61uN6RJAV5XkEdGLqOqpS1rdFGOoamZltvoi9UlBZ4L/FKxycczA9QRvgxdV1zMJ+TZSnr9QprLVlRYRLNQXoYyXKKEBQmSSlHCaPRpxwfHXJycsT795fc3N/RND23dw+SEtL2dH0v1jnxdTVaXA/27LBd0VIaFTzax2h5kzDOC8ZVxdPTJ6zv7wmpODY+efKE09NTkrhq3olTu74jWMEbm1o6qflsRvCe2cGMoR9Y1UuqasSbN29YrZbc3d7y4vlzmqbh+PiY2WyG1nB5eYk2EvA5m85k/NtuydKM09NT3r15x2g04vz8nPlkIn7v8WDM0jwGYYpi/1/8xb8kzwsm4wlZVtA0LUEplssV8+kUpRR5lmOiT37btZJs0w5cXK/55W/f8P27e9aNw0ZROUrhY+F3RE2os5RK06mETqXSSSBxY6+ulry+uuWr1xd89smCj15u8Hi++OILjBL293g8YTqd471ntalZLh/wzjGdjWOklmNcVmRJStNsWbX9Pg3GmISyqmRsNUaMFdNk7yPVbGtWyyXLxZJ+6KPFiliEizuske5ZG7wX5YT3TuhGbhASsBFMKzGJsOdjd+5D3HVHYmgIjrZVdM0O5/sRi9TjvbobSGL7hwDkSoPRitSoCIaLPUumIUORRetgjY8gtJIVeuygdvIWwh70khVs2GEhgkGFXd5exL92di27ixCExrCjcWZpSgiyxbhte+7v75nNppyenvD06TnOOrbbzZ7g1vei6u/7/nc2fFIj47jpZUx1TnzMnRWSa5pkDF6AedfLllGlGu8Cdb0mz3buCYYszyiKXEim0U5mMpnQtS1XV1c8ffacyWRCkueYNJqZCQ1LbHqjIX5Ti4F/VclqGRSDc+gs4enZMYcHM148O8d5T9f1vH79lqbraOqGm9tblssV26YRL6cgZFblhTejEAaxUjEnLUoegguowWKA25tr+rrhoKwYj8eSnOIcSmv6YWCwjrIsRUsZdZLv3r8jTRKyNGU6m+G8jGBFUUTd3cB8NiVNFMuHBc7aSB3IWa5WZHnO7e0t/+ov/5JvvvmGJ+fnHBwc8Mknn/Hdd68Zj0d8+ulLxmWOHXqyRNF1MQklhrYqFAfzA87OnrDdNqR5SdMP6CTFRNfVqhwz2F621lkqS5imZb0deHdV83f/+A3vblYsmwFvUrxW4tQR1GOBig4bXnk2Q4czKQMeVDQlMjvdg2LZOV7f3HO/bQluYLtuWdw88PTZU15+/LEQTbOM2XzKbDahaxpW65VwEo0hz1JG4zFKKdqmwyRix9MPA3ZoyVIRIdf1lq5rxfZYG/Ce5XJJ27VxIytxbnmRMx6NKLKUQKBvWpwbJL3HRUsjN1BmCaMyRynBSttWmPLCg5KpR4iismhzyqNwpOnvNz0/UpHSu+SVOP4Yo0li5I1WYrtiEEB8p+nLFKRxe7dDQMTzRaQvEtog1VkijtjLXXZdVPCP9ITffzxaqUfJS/gw0ioSPXeFVYsp3nq9Yr1ekRcZJ8fHHB4cRta5+DPnhazMd+S6EAmdYv8R9m4PNs7sNq58ZYST7d+Df8BZz2wypShz2Xz0rRTpyLot8oKmqUXnNVjSNOfs7AnaJNze3eKB6cGBnIBZVPor2VSu64blYsloPOLk+AQTsSLvPanSzCYjtEkZrON4OABgvdlyenSEC56maWNH1fGwXMULL3J0BrnIjDHc3d1Rb9tIwvRs65blYgkonhwfU40qLt68pawK8jInK0u8CmR5gXfRccJZGRa85/b2hqapUSqQFxlt1wjLOU2pypL7uzuqouTLL3/DbDqm7zreX1zsE01W6xUff/wxRmsWyyUoRb3dUpYlv/nNb3n69DwGtRqWywVaTXBWkolHR3OWiwXzakbbtsxmcwIwmk5Aa2zfU+TpnpVNLDLD4AhhYLNa0TQ1X3/7jv/lH77jzeWtLBYE7MCDvP8+Rr0puea1kuu962rBtKI8Yg+beE1QCqcV6XjEzWJJt9qyWW65vbrh07s7NtstL18+58nTc2ykAhEck+lEeIC75Q+Sj/fsxXMODqbc316xeLgTvLRr2a5XcQOYoLUS25rNhiRJmE7HMuqGQJ5l5HkS04blFkpjsfJeVBmJ1qRVhQsBpSUlu2m66M+fk6YZLni6vhO7oBCzB4CgzH5L/KMVqf06fbe4D+KLnBgTPaFEXGyUGNcZECO8EBNh4smMMtI9SYQogUE6KS/gXnAudk2DpBZD7L2kKH5YqJTaAfc7lJv/AhgX2AU4ENtOhVisdF3Hu/fvubq6oihLXjx/ztHJCc45qmLNYrlgvSPOIfo28cyC4J0wd5uGttlie7H7UDGjb+gtXfy3w0PBmZy3+GFAGxiPx/jIYE9iUrGOrONyNOL46BgHXF1e4VGMxlORSyi170yGYaBtGom/9p4sk1AAFLS1j9jIoz3vaFSKpYf3HB/NMCZhu6mp6ybSCRRuEDvmoZfH9vCw4N3bd1xf37Jab0hVYFRkAqbanpQCHQHuw9MjysMpnbM4PxAGL6O0lvV419QMXcNicU+WJ5yeHuGcJ81SqrLi7vYOFQLv37/jcD7HGMXQ9Xz68iXfRnzy05NP2XlFLRcLzs/P+Zu/+RtOTk5Ik4zRqGI6HqPjFvHo8IiLd28p85IQlwY+BI6OjpnMZozGEzHrU4oskzit4AM6yWi6nrZrSY1ms225vLnjzQ9v+NtffcuXtzU9iEcZCh/840AAIo1RYJQniZessZbMGILW4rdGbNSVfKHSnixP2DYNKmg2m5Yf3l7Q20ECM5SldS2zg0NOTp+wfFhwe3dHWRRURUlVVvRdL5w6N8ScASQLUWmxJNZzhqFnGDq6rsP3fdx6pyLhgrhgEBncbnzVRmOSTEIzlCEExTDY/XbXO4cb7D7vcedK2zuHDRZRbUgxDsBgHVpZ/pCPP7yTUjtgOmbjKUWaGlItBSqLDgj7BF383rN8ly4MwhpHeZQSHEo6qAHlbeykXFRZy/gW2H3tf+1hRfKZCWin8ErtcamwB9jll4sFRsX2WGmDeKHCZrPh66+/ZjyZ8PzZM549f8bR8RE/vHnNar3G1w1B+Tj9hb3vtiS3NHSttMKJEnA5MQZSQ9M0LJfiCZ9nKQEvJ7OXTDsdAyJcTNIQC+GO1XqDC4GTsyfkWcbD3T1FWYDWrDcbQPAnlGKzEa8gyT1LowOix8aY9CTJ5HUyCXle7DlcWism0zGjcbkvZq4XgWnXtqyWS7RybNZLtpslGkWeJKz7jizPCN5y9f49WgU2zYZ+aLl+/5bToz+lTMWi2bY1vfOkWcbQt7x5+wNNu+WTn/wJKOn+yqqiazs22410qV5CPG6v7xiGjm0NH330Eff39xwdH7NaLnn37h0/+fxzijzn/MkT7u7u+OKLn3FxccHBz37K+fkTLv3AdruWbdfQUdd13BQ7zs/POTw+ou17tEnwKIqqZOg6hq5HaUW7bciLguViwc3dkldvLvnlr7/hzc2Czont704bKigUkiUZN9xGgUFhghHKiDHkRoI1nA746H7qvFxXOlXoGDQCKYMK1IPl9n6N/+03uEE8pU6eNhzNDjg9PKaua9brJZvVglElljwmkeDZfuhJspSyKNm6gfVySVPX9F2D7fvo5+WFMmIMw9BGLNgy9BJuEowiaEOwbr9ECvGes9H3SqYNSbgWEbGl7zsiiR2MEkwvQjEBhU4S9I9N5kyNEg1SiMVKQ2ogj5hT8jtFSkfjOmmCDY/BoToa5Cks3vdiiLcf9aSLCk7Qf/d7QP3vj3tKqUiF2P/F7/x5t5HbA+1h1wUKD0jL+mZfuLy3LBYD9XbD1eUVH798yWef/oT1es319TWr5ZK2rrGhwwcdJQAJIWj6TkITksgRG5Sl78EOLoqXLdPJhDLPCd4yWIcL0A5CJO26lqHro42r4fTJmUgqQuBgPgdE55amGePJBJRmtV7Ttg1ZnpMmKRM7YTafk2cijO17G/2tJNY8y0sSk+55YjbqCouijJYzCj1WdG0nxEBjUIp9XlqSJGy3S46PDjk8POLh4QEQS1kpjANPT49JVWBotnunTrHG7fjtl7/h4GDO/GgmnKQ0l1W2EiV9iDq46XjEavnA0fEhtu8Zj8f8+re/5eLiPfd3t5w/ecLQt9zfXPPixQs+/eQls+mEk6MjfvbF5/RdC96xXi2ZjIRk+HB3R9u0mDTn5OwJ48mULC9xoaeoRljnpP9XCcaEmPjjqdc169WWy+sHfvXVa76/XrNsIcSbbkfY2Em8MqPRWqxNsjjyTCdzskK6FQX0fctmu6KJXkvd0GGtJ8uNkJ2VUDAsnsEr3LZDeXj76j1209E1HdrB5599wXg8Znx8wqauWS7W2M4ymUwwSULqM3xwKDw+RD+zfhDfrKqUzsYNDG3HcrmUz1UiY1KIvY5rhS6zcwhJdJyotCLPcwY1PBYp5cE5TKJIUXgdYuSZipfh48a0yBKK/EeWxWRpBiGgg5wOWgWMUXF758XAjhhKEKTbQhGDOQG8dFGRrOm9tKXCIJd1fASi9gVl97HX0IUP/J92tQjx49mFf+46K61lu8IOo+J3N5Q7e1Rhk+t4k0Yh8NCzWDzwj8sFk/GUF88/4rNPP+Ph/p6Ld+/YrJcYK5FGQz/QZBlJmkZwcNg7jA5K1v1N2zB0rZxCo0hxUIGDgwPA7zujXc7earXh8uKC0yfnUWBbiyjbCLeq73sms7mERW4FAC1KibYa+p5GNcznc6bTOXlRPJ5aWu91WD6+7lkudIAkSem7jr5t0BqqKqfZCLaTmITECLlxNh5jkoyriwusdzTDIMkivXSE68U1iRrQPImjjzDVL26uCcFxdnZG0zcCFSQpm/UmLi5yijyPHXDg5OQEa3veP9xzf3fDb3/za5q2Jc9ymqbBWcfN7Q11U5MmKZ/95DOqckySJNhesV6vmE6ndK0A8UPfk6YpaM3BwYG8LiZBJ568KAltF8M+xMqHEOi7nqbtuby65R9/+WveXN+zsprBJDsGTjyIA1WqSCMFJitKsnLM6elznr74lJeffMH44Fg6C29p6hV3txd8+82vWdxf0NYrmo1jPC7RzpEniYyDwYtrgvOs6paj0Yi7qzuUAdf10A988dOfMj+YMxtPyE3OdlvTD0LhsTs4YLsRg76ixAB5mjAMnRw8OsfnheRYRurN0Pf0rSBteXRmCEEKN16Y73iFyjV5mu0FGm6QlOxhGPa+bLCrBTt8WaAipRXe/8jjXpak0XBA8Ca965JUiEC53//9HivaEy49BCscGxwhzswhCNvb252R3s7K18et3iPDPMS/2xWwHfyklLDdxdROg9egA8GLpsx5KQi70ubCIyF015jJdlDkKESw3SOj2Gaz5ssvf0NZVpyenPLJZ5+wXq24ePuWZb/AaGELJ1mKt7Ldc0MfZ3CJefIeuq5ntVrhvSVJdCTctaRJGiUsHqNMtCtpuL655vryEu89Dw8PnD05ZzIeUZTiDFBvxQzvk49fMjhLXbfc3d6jlGI0lnjr8XjK+bNnQn+ICv28KB7fH8AkiTDn+0HiwruGvl2zXS1ZLe/Yrpd0bQsu0Nuek5NT7u7u6XtLpzyqyhg6S9tZFos1b9+8FveLIifPCwiwWgsW9fz5C3rbMxqJJXLbNOR5TpoY2rZmNptRrzd0TcOXv/oF1ajk4eEOpRTnZ084Pj5BJ5rLqyu++OILBjuQpRlt02D7AUrpSMuypK03TCcTLi/eRR80RWctz56fM5/PyfMSpQx5XkrUlzG0XS8Gb0rttYdv3lzwv/zDP/Dt+wse6p4mgtw6IN2TEtF8mShKY6hGE06ffcSzlz/h5OwjiskR1eyIYjzHBVmehLxgqhN+kiZcvSu4v35HnSlGVUHd9IzyDB8GlNP00d+sHhyXd7ecHx3wcLegLHLemEBeGAIfk2Ul1iqqaox1A5vNir7rYxPQRxnZgO07QvCR2hMgEbcSUTGE6BsnLghFXgoE4N3eZ8pbEfvrCBeE6P6gApAocp2QGC0YmvcQuVE7Eb53kp7tIsXoRy1SwoOSCCujxDdI6AcOHWx01ZRC83gLxK9FfM7Ft8bibQ/eYd0QJSkSEuo/oCCEEN0Qdh8R15J0ZPYnGSo6YpodsVTsJQR8j3t7JeprHyUULsprdi82CoLekUrl+4tcZhf6ICB1vdlQjSqeP3vGH/3xn3B1ccHr71/TtA1p1hKsizwu6RqN0qSJKL6zXOb+frBs6w6FI4kkuVFZMpvNMXFITpOMQODm9pbNZo0yitWyxDnhRx0dHbFeb1FaRkAPpGnG6ekZTdPsQzE3mw3/9E//RFGWPDk/5zBa4u4wRYCuldh2ay1D29I3G5Z316wXD9zf3nB//0BZlISZYb2tMVmGdSILediu0TYlND3TcixUi23NxcUFs9mUbtvivGe13TI/PgECWSbi6CaC/UmiuL25lk7dOb799lv+89/+R+bzCccnh5TlOcNgGY86nBPCcJaJ/bDEWz1htZAb0nvPwXyO7QeMNtiY8LzZrKmbhmcvPuL58+eMRiPgEcTdbGtA0fY9SZqKg0S94eHuni+//Y6vXr9l0Xb0XoM2EZeVm6dINFWmKNPAJMt4+dELRkfHWNdzc3+Bqdckq3t0WgKGwVqctXRtTRjEw/709BQ7K/FDR9/dk2ca6zXaKZSNmYMEVk3DcDUwa3J0pkgLw3L1wNVlweHRGc4p7u4WnBwfo+PWc7N6wLlOIs/cgAvipaZU5L/F+ydLI2juHIlJ6Duh4nRdK0XNDdhB4tSUltg4H9OBPtzCy+QS9uNd/N+eOhTUjoTN76k4foQilcRVvw7Ci0o/KFTa+Yg9PYYUqMhmDgRUiGRNa3FWyJLKC5OcWBh2NzchPD6h38ek9pPg72FTWkeO7+7fNcabWCzCXrSLc7LxjfE9WgnQrqNhnXMgku/IYI/dlYyU4gS62Wz46uuvODs+5eOPPuLJ+VN++5vf8ObND2wWS5EbdG0kuUlx3slITJqSGkkl2Z1WisB6vWG5XImZXzx1sjxnPptRt+2+6wBJG1kthQLQdpKVNhpPaLotSidopdlstrx584bEpCKn6EfcGcgTGI2qmGAsF0hALtrtZsN2taSv1zTrBYv7a64u39N1im09sFw3KJPy/du35GVJV69xiBWy8nIYXF5eARVpkgihshxRNw2T2QxjtDDPx2M5TV1gPJ6wWa+ZTufU2y3Be56cnfK//z/872jrDYnRbLYr7u7u9g4AZZkLY3y1xFrLL37xT/zk08+YTsf0bUPftayXK8muUzAajYRukksCSlFUaB01lV7M3Oq2IcsE1xv6nma9oq+33Nzd8+qHdyzangHZkBni4UwgN5pRqqgyRaoceaoZ+oab6/f02hBMik9yMAXWaYY+0HXCFxqGjsw4qjxwMh8xGY1pt3KfJMpRZWCsJCq1AfpE0VmRmDSLHp0ljMYV799dMptIlPrJyRNSM7B4WDLYliRJGY/H1E2gbTYoBEdKIpDtvdsfyj667BqtUbGrUkrInMPQkwSDLnX0zpKsxKauadpW7iUtqdph56PmvUxHWkucGY8rLOKm70d3QdiHeHpFqrRk5AUfRz2FCo9ZLjqOhT5iUCpYghd2qnd93OBEjGo31u3SZvi9IhXBbWD/gu4btYAA5MRCFYRqsEtwUQSGEPauhLsv1XuMSkY9CJFlHh6jsYJYBgu+JUxbE2dz7+Hy5pqHhweeP3/OX/7lX/Lpp5/x21/9iov3b+naRpTiQ4cKNhJKxQiwqiqybBqN9Ya4RBDsx2gtIs3linq7hRhHb7sOazQtgel0QpqllNWYbujZbLZUoxFaQbOtsc6SlwVpmjB0Lcm4QmHZLO+4cA1pksbY7AKTiNhao3Ftg++22GaD72pSrRmXFdfXF2w2A4M3LJZrVnXNeDphUW8kzMIFEm1wPlDXNU0N5vSEut6i0ZydnZFGoqyJ40Nd1+RpThtjrXSMUErwGB1otxvev32LSRS3Nzc47zk6POLu5obReMThfB5vnoHvvvuOrmuZTidURU7fNhG/k7AM1znyoqBre05OzkjTnLYfyAuR6VgvMU5N05CmGX1b4/qGu9sb/v4Xv+Ti7oGeBK81WHHwyA3kRpMniioRn/5EB4Lv6Zo1vUmog6a1gbr3NEOgGzzBwtBLgQaH1h4VBvQXnzCfnTFihA5OxPiJOIWIZ4AoD7zSDC6Aciw2LQ+rBu8uOJgekaYlq2zFbHIIQeG3PdoYsqrEJJ5Ee8FLXYBMEpuHvsfaXoDyuPDZRa4H7xmGuMCyDuftHlSXe08OX20E5ySEvf+VD15Sa4AkSwWg/4AULR9+b0fzoxUpY4zEUGkVT5MdiL5bIMmaNUQMKhAISugFOuzzi4W8GXykH8iIt2OUEyUZv1OgeCwwv9tAeSkwkVgakBnZhyiE9YjOaO+6GVNjtCGYsP8eu4IXIrdjl9oBGq18zBiTAkOaYuKP1Giss7y/eMfi4Z7nz17wP/yv/i1fff1bXr36hu1qRaLHYHvxOffi7902DdYaqjIny8U0b7tZ78eY6WxGlkqkk9Iwqiq8l6CFAGzWK4IXT6+irEgSw83NNaCYTueMspLVek1RZGSJMNNXy3smkxFdC2QpoyqXLs8PYD2Ddwxtje97XC+eSUZryqrk+fNnLJYd33z/llXbgNHUXSuEQideYLLxsSSpuB9Mp1PZoAax9TmezyEx6CwlBE9iDH3f0jUtWZaxWa9QCrZdx/3dDVWRURQF2+2K6USkKf3Q8/z5U95dXPHpZ58xHo8Z+p7z8/N9XtxsMuX+/h43WIqioG0attua7abm8PCIw8NDsixnU9dkezxSElf6Tp6z8p7VcsU//eLXvHr9jm3vsDqN0EAg09G8MZVfCRYTEEA5FYLkMFi2raO2irpzbNuewUZYY+feGq/9ssz44c07RrniydFESLkxRSmNBtlyR+3AFI8LsOktr99f8fzokFffvJIAjbzAkET1hDDrg7N4OwCBLE1Ai8e+90qWDLan67q9F7s20cvfSAq3DzH3ERmfRUcrhE9jNGVVUAa5jm3fi4+VHfZW4LtQYQhRB/o4IeldwfuxipSMZR9k48W5U0a7CGMrJcRMRIAbsPgwCO/DDXs3yV1ns5e7RF8mfr847XlSu8ewQ7tkhNstuHffZ8eLUh8Unn2n5QUc10aTqIRdeEOIALrfeZDDfgzz2qO8iukvAiAmaUKapLKt0AadaAZrefXdK27vJrx48YKnz8/58je/5u7ykoQCozR26KibGjt4/GBZDz3VqBAjsDRjs91yc3tL33cUWU5RSqTQ4uFebD+C58nTpxgjbpXDZoNZr0myjOVqxWq1RmnDbD7j6PCINM3AKK6vLwiup+9r1OEhWSJs66HvJa3WJBIq0fVYa+l7zzDshEWKyWxG67c4rZnMpqw2WwbrMTpFq4AdesbjEYXxnJ/OOT4U/R4BdJIyPZhTjoSVvGlakizIuBkCeZHS1DVJ9I+62Wwo84LgHHVdY/uB+XzOZrNhEoNOi4cFQy8/s20M0+lUusIYwJAkCZthYLsVJn9TNzw5f8bR0TFFNSYvC+quZxgcbd8TUOK9jaFe1fRdy+u373n1+j3365agC5IkxzmLVgpjFCYBbWSjnRixtzakJGkBJsMPYgjZtJa67Rns44QhHQT7bt0Dq03N9e0Do8IQtEFpRaLFVNEpRTDC51NBoXygd4oBxaruuddriIqHNCn4+OMcN/S0/Za+26Cx9F1D04gUJjWJuGQo4RGqiNPu7jOt5XoG9sTgoAIaTZYlOAXD4ASzjcsFE6eAMAicI3UhUoBUJLsikXeSji335R+aYvyHY1IqSCe1izYP8Vd005SuZkfA9DLiBWGQ76J5vBUWbPABvDge+BhKsC9Qu20gv0ceD8Q15iOF4MN/F12b33/9f5EEqh7DGfwHRTJE0O+RniCFVocYsBBP0d3qHoSSsZPKoMUSdb1e8etfLzg7O+Ev/vmfc3t1yde//Q2b1QpFkGikLBMPdDfICaZgNpuRFzk6MVy8f8/DcsloKHh6/oQ0TWjaBoJns1pSjETuIsQ6GZOfnj9hNp/y5u1b3r59zXLxwPGJhEI615OnKZPxlPPz5ySJBD3meSHupd5jrY8yhpyhE8Oy65trusFjCsNyvWI0GfP+2+9J8wIdINWaYeg5nk8xrmda5cxnY54/PSdNxKA/yXLyLKPe1mybhnI8JtGarhV8LY2r8IPZIWWekz455+H+hvvbGybjMclsQpqmFEXBarPmaDLli88/Z7FcibndwQHHWuMGy9D1sg6P18Xt7Q1Floo2c7Ph2bMX4ktvnfB7ItPdxdw4ZyXB54c37/nmu/e8v1thgybNCykmg8ckJpq3BbQO6EQ8sNJEoTCoNMPreCNbxJO8t0Lw/ID4ubu2A4G26zHec333wHwqj+uR2+dJFTIWJTpe/jEyisDgLU3XwWREvW757ptvmY0njEYVg2tpmi3OdnjfR3/+QRJofPoBHiRFxjqHjmOR1nq/OQ8fbOtDEK6chHsGQpyGrI1nto5W1FHpsAfUVSAxmtxIRJmJ8MmPnruXeIcOgSTsAj1jkdh1RexcNe0egyKCZ8FFvGkvygOCjSJiv2cZB+8joCXd0AfjK8JWjz9v3ymxz2GQfxInAD7Ax/Yf8YUWqY203C4QxcK7jlBeWL+D91T0Xw9KZlot8ocdCdIHIyvrOGpqIyPv5btL1vcPfPLyI/77f/Ov+cUv/pF3b97KwiFJyfNKSHztlqYV8HEyHsvW8cmTqBuUWHGtFdPZhLLIWC7uuV88MB5PMSZFJwbd92w2G9q248npKcnTZ+RFwWgsG6z5bI5COD+XV9d7vO7J2QllUdLFTDaU4uL9BQ9396zXa9Is5Wg64svvfuD66paHTRcxumTPbysSw8dPz6iSQJlI0IYG2rphcjImKPGOClpzfHK6N85TITzGeTvPZrWgMYaubXm4u8cOls1qzeHRnETLpjLPc96+fcOLFy+ZTWA8HuH6gclozGq1ZOh7jIbl4h5jDMeHhwTvSU3GZDZnNp2jTMbgPUmWM7QiuF3cLyWmq2m5vrnhn37zNb/66ntaq0mKAhJD23ZoHWSVroXzI+4f0gl474nbF5EqWUdr7R7TwatIVA4SDBJx0hCvRUeg6y1XN3fkyomcDB9lZtHGSGtIJXFpcIHByjjddAPrTc18IlKgq4u3nJ2ekBUpzg7U9ZYQBrJEk5lEZGeDw7m41or+aVmaRmKzOJFoFRNdoodYcNHNNj5HwXBlgnLe4ntRX/gPt++BaP8jzUFZleRZJpScSBj+UYuU8R2SLEyclOMj2ZWInWldGCAI9iRd0l6hJG/KrhvxIk8IQTqv/S7Qex7LRKwv8YnvCJsR8Y6FbNcR7X+M/PZ7G8AP//t3yKGx2odonBS8j1HyO/BJMs6Uetxe7m1ktMNoi07iSj+C7DpN6duBr778kul8wh/9yZ/w4tlHfPfqFevFguDBZAmj0Zgsz+j7Dm00Z6enMUF3tDclS1NhfTtnOT4+RCU5SZrR1C0PyyXL1Zqjw0OSLGX18MBoPGa1WrFdl2iTslwuOT054fT0VFwe4oXRti33d/d465hMxngfODw65Oz0lM1mw+vX33N//8B4NOLsRHO/fEOal/SDBFFWeUaVGWyz4vTjF5zMJ8ymY8pqQjmeSOLLwwNFNQItI2VQms22xkdCoASwKqqyYvHwQNd1HB4e8fr7V7RtK8VXG4pBiJhlWWKt5fDwgLpuxC+prpmOxrRdu38/kyShb1vevX3Ds6cvePLkLFoE270bpELR1A19Jx5NV1c3/PD+kl/+5hWLuiUpcrKiohkGnBVum9aPK3UTRxVnLUWZoZOM1jvaesuqdtRNG1f9Cje4x+vUPVrh7OkxXlJbmrZDJ9Kl+R3mqxHMKwSC0rhU0zowPsStqoQxbDZrQjiirpdcXnUcHx/hhp6u2eK9RRUZSgXBjYYBpSWpRsdQ0t09YgeLQ0ZbpTVq52EfF2Fd24oBZojBWkHSmaxz7FK/5e55vL98EEdYQojBo518/u/fo/+Vjz+8SAULxJy7HQZFTPn1nhAsqBhiEMXCBMvOiiV8YNEb8JFj4fabPB+fjFRyeYJ78iZa3s9YlVGI7g4lPFFi3HSIn+Mfx8YQ9YExh4ZdOKfzLnZvcTD0MXcvFieldhFe4jnqnRBXMSoKN/2+4EZnmTjqxjc3CSinWS1E/nB6fMRf/uW/5Prikq+/+jLyURRZKtatfaQTzA8PUFpz13UkxuCs5/BwRlUWtDE5WaKOhIx3enTE0cGhqOitQ2EYFzmJSaiqEedPntB1PU3XMZqMBQz1jnrb0g+yeeytuCkak9K0Hd+/fs1ytWC1rsmyMQ+39ygnFiRKKZ6cHnM0HTHONL7fMp9UVFVJmol18sHhHBcc48mErndkWYqLoRp5notbg/UUVc52G0Mpsox6s2Ez9GyWSxSB9+/eMZvNJECgHJHHiKuyrGjqBhPz7Zq2ZjIZ0/e9eHI7yzBIWKYikJhEdJtA13dkOTgc23qLD462a7m4uOLv/+GXPGxqQpKRlSNZn3dia71TMxBchF9ljZ8qZMxMEurO0VtHO/S0fY/zCtCgNP/m3/wbXv/wmrdv34rdqQr4CKILfzCGx8ZD0AaHUY+cQ62kSzVa0oq0SuQ+itxDpWFbr9G6Z7PxONcyKiu0D9h+YDv05FkW3VAHAoMIjlO/z8LTSkitwjWMhWbn3xa34HIvPWYR/O6Sa0eu1nL/6RDvFbk327bFxXg5Saf5kYuU9rL52jGzldpZBvuIQcVC4ARvwdt9exjsLs/u0e5k92v3JHd4z++g//K0H4lhsP8376UlVbu2Oew6sRj0GbkaAvA9bhCddzH2x+0fA+w6PBe7KPUBfrD7eQGLJdEpOr4HfPDmhF2HB4S4ylXGYHzA9wPv376jb1q++Pxzzp6c8Otf/ZLFwwJjIMsLlDbR+3vg6PAQYzTb9RqCZ1RNKIuEwVkWyyUhIq8qjq1KG46PTiIQDFU14v7+gdu7WzAyhj7cL7i6vCTPckajCm8d9/f3kVxZ4QZLmibYwXJx+Z63737g8PCEgEh+Tk5OWLx5x/+btj/7sSzL0vyw3x7OfCcbfYg5IivHqu6qarKJlroFQi8C/1FCakjUm5qQBIiA0IJAdrO7qrKqsjIyYw4fzM3szmfcgx7WvtciKULMEoMGWLh7mLnbHc5Ze61vfYMPE2MbmKynWVzz/sc/4fLykvlsTt3UxBg5Ho9s93vmy0vyskIlMFYpRdQKpWyKLy/FB79p6Lue29sbhq6lqiu0ilhruLi6TDa3sFws2W63HA+HVDDERDC4kb7tIOnJtMr5zd//LRfzBfOmEY/uKBJgbUV+1HadkBX7gS+/+JLf//4L3ry9I0Yoy4qskMRllcAYFWWJwgkCUIrMWMpMQ5RACTGOTZIWlw6sVND++m/+hsPhcL6R/2BSSAXwxMRW1mARWoNVihgUMWq0O4n35e+ECD56tMloZg3jOPL4eMQY2O33LOZzFrN5IiM7hm6iKDKKspKAT+ewmUljrEn39UkiJvdYON+PKVcwk9clnkjbQZ6P0hpjbLIaPk0+ovBQweNjZBwnxnHi5Kf+43dSqfqr8wufQPMYUkFKjHI/STsZE0nzByrpU+dz+gxeKAPSzeikbZIWNnihB8gK8ySzeRrPROryBJ6HRDU4dzc+dWfhyf/GpeLknWwmgvvB2AdpQwgoRUh6QAEO9XkLEkJIzoP/Ix/xVKhiGgsEWzNJSf74eM9/+I97fvHLn/G//lf/kr/729/w/fffEmNEG7noh75nM47c3j6nn815eLiXdtlKTp5gYsL9ef7iOXkxo64X6LLkk48/5di2DH3PbVmhtHgAGWPIq4r9dkduM5qmZhwGrm9ukordMwySEPL2rcQynV9L7xn7gdFPmDCh40Sh4OeffsT1hZgHVnVNXhScLGP7fjyTJ9vjkYiimc+weU4/jOz3e+p0o1QJXA/O4dzEfrfDaku+WDJNYyJfao7HlrKsaJqGru8osgyVIpUkuNVT5mLvO3nHxWpJXVSykZomQtAEpfBO0l0iimEYeLx/YLfd8/r1a5TSaKPOrhEg/Lvz4XP+D+cDKnjRq40uEnT+ZOVz+pYQ0Maw3+3wQaLYYqLOnEjPp0vn9O9n1lJai40eQ5QiFTTqB/gpMbXvKjCfN1RVxdBtqQrD8dDR9gO7fUtVbpk1s5QROYjVTy35eRkQXCBqicMySiRlfnK4tCmPaRGmT0LhROp2gTQ38PT/reFsWsnp+Wmxx0mJTFInTobVf9zHH12kZlV9fjBGqXQ/+mTF8gNsKYji3zuXgjbFBeAUYRNSoXDKofVpXHKpCIiR3OScvNFKM0xTshOTm9/aNEP/oEqcOrEYTvHlye3Sy5+Dc0zTwDhJMXWjO496p8H1ad+SOrPw1LmJ7MaKK8APfnD8g9+dPk+YmYZgAOGIkZTou92Rf/fv/jtevHyPX/7qz3jvww/421//Lbv9huAdeVkR0836/NkteZGz3a5xIaCMYr5oGAYBnperFbPlc6aguLi+hCynWebk9cg0OeqmJPhAlhnKuiY3IolZrx+J0ZNlOZv1Wtwwx5Gu67i4uKCqSlarJdMY+Oab1/RtyzB5Pv3wBbO65MXVnI8/fI/VcsFiuWSYRrq2Y3IGYzNm8yXKaImI0posL1AKqrIkxMhsNiPThizL2Dw+Mo6SrHv/7h27zYYsF5pFlue4ybFcLiVeaxop8iIlB3tsbdnvdvI+KikIxMjhsOfi4oJX33zLarHE5BW6iExRMfhRbnZtWa/XfP/9K7788iv6fhQAGIULDhOkq1BaExUElQqJOl1vcsBlWUaZlfgIRwfTKFtLYy2TC2kKkAPXGNFsBiVJK0LyiKDEaSC3BWWhqKucUkdUmLAxABaDxmnPMHqUFpxXEcmznOurK5FMZQUXqyVF1bF79YY37zYYbWialro6JSVpphhZLeZYWxCRLktFiFYe+yk3QORScqhrLZQE72Rc9JNLOLQskmJy3QxeEmWUOt0HYJWRYp8jbPtxRLjTP3In1dQ1pBdVyTQsxMk4idVJoppFFSWVIrMiO0nYjUJWj0qJ7MMoc0KNUpWWbugcG5XGxGHsOe73jOOA90GsTycRw06TxKG7NN6dxMnSHUhB8kkB79z0VDC9S35VnLV5JwrFH+BjaZoLOsWiG02MJo2SihBMKohicREUad180sbJtlFMwSackxuhO7b8w2/3vHn7ll/+6lf8Z//in/Pb3/2Or776Fu9HqjzH6si+PTBfLSmagv1hS5VnNAuLc4rLyytJObl5iVeG4/HAoetSjFXNsVsTj510t3sBf3fbLVormqbh/v4du92Opq5FSxacYEXeM2sadtsDwcsJWpc5daX57JMPuLm+YDUvmdWVjNxGMRx6Mdf3gaqZyeuBXLCry0vKuuZwbOXmHeWSW283XF9ecWxbqqpgmkaU0Zg8o+9asqIgzy3L1TLhF+KT/vDwwHKxoKlrNo8bIJ4dRUN05EXB4+MDdVVS1dKtmcOOMgqC6seREOFxu+dwOPL69Vve3r1jcGPS8onVbQgSEqB0KiU/pKj8gIfnnMNWBZnJmAZPjqLwmnyQ3EKd/v6JeBxPJOSTMDdI/FNdZFRFTlNa6jInixPhKc4HpbQEbUaP1oPYqujIfDY/mwpe314xWzQMG8/kI/t+wAV4bAeK8kiRZeS5pcwz2ilwvVpSWE1uVGKde7LMoUxMiU8apWU89sGJDfEo5GSfRuGzBtAgxTdGsbE541wi1tZRNopucoyjS7jV/+g88v9/kdpsNlI8vD9XUGl3HCFMQjsgnDU7ciO75Gt8cr8SGkEInM2yTszbiOdkq6sSE1WjqKqSuqxRWrNaLLh59uw8wqhkPdKmLU972NEdO46HPfv9jrHrGF1knMCd0ndd6pLUyVLvCbwMyWtc/L1PmIIUmSlRH06MaR+sOD2mSuaNJViPtT6NCzptQNKZmUbGM7+JwHb3lv/2v33k9vY5f/pP/5KbZ+/x+W9/w379DhUDbnRM2w3LiyUX1XMOhx6thEgajMGUFd3YUc8WXF6u2G13IsadzZk1NX7yHNs9RVHgxoGmLDkejiJZMRqjNU1Vs10/sD+Ia4CbRsq8QEV4uL/n4WGD0RnL1QXdccvqT94XRnRuePP9G3aJLT+bz6XIWMXoR9wYybKCqJB8wbJgTCnYwqyXWHObZzyu1+do9oeHBy6urpnPGxmNYmQYBspKDoemqlNgqJiy5XlO74Vh3g2t/D+bUeYlYyduFH3KjGyqmrZt+ebtW7769jXfv7njzes70UBqkUHFcTqPNyFIF20yK1uxADaBwRHx3ypy8U13UcJSS61ptKb3peBwMa3wz/SWRHVJMhQIlLlmURfMi4wqtxSZJYsRZQqZLEJaGJ3+HcFFsFpTWkt7OFKXlhAgy0uU1rjgmRQMUegnapwoMvES00rx6t2GeVVyMau5Xi2pywKFdN15oZnPKmaz+vwehCGeQ0GtPXn1n0Y9eUzeOdwk1KLTmOG8ZxxH6b5cJPqIRmyRxJP/RyxSMsmIp/mZBpAG6afNoycmd0yi6Hm1irLZSbasZ5A8bQlQkRidbCmcbNgigkl55wTUIyWaeGmTldHJj8hSFgVVI5jIcrXi5Xvvk9kcpWEcRdu2ftzw+Lhm/fjI3d0dQ9fKaKggRp+6oR9wtYipNZfneGpK3eRSQTKYGEjJphKFbQMxGCl0URONJiPI+jgNlVpZlPLJ49oRcXg/8Or173hYv+FXv/zP+Ms//3O++fpLvvn6S/qhw+J5eNxwcXnJs+cvGUa50YdpYoqRpqzwfqIsZlxcXvD48Eh72Cdui+gqu6Owjf00Ya1NoZ3QJisa7wPXV5dsNmu+//Zbrq6uaeo5NzfXXFxcoKJisbqgntfMmoaH+3d888033N29ZbVacnl5iQsTH338CVFLOvFieUGel2y2G6qqYb5cimfVKH5WfddTlaVY1VxcUDY1x2PLfLlAWU1elomgG7FRFAnOeTJjqatagi21xXtPVVdnbtfjwz3Pnj2jPRx58/o1WiuWiyXHccKPI8d+YLNZ8/D4yOvXb/FTSDS8U/cr16gwDBQ66rPw2AWPx8hWLsECpy5mCg5jCwpjKTXMmoK+H5jGE/hOmvpEtiWHcECbyLKWYjEvMgoDuVVYNNGJTjRpJoheMha7vkPjKTNDDI5xCFyublhdXPHivQ+4vL1hUobhHz7nzcOW0Qt+O0VP5lPnEyOHfs9237LZt1wu5tRVQVUVlN4QtdzHEgSK2Lw4J6GfNkebJ+DDB0ffd8nux51bzeB8sip2TJNPlBOx7nbBw/THmZz/8WTOhPifEUEgqkhMdP2I6J7EUjWkLkTWr0oFvEKifZQkURglqREyCrrkIZWCA6MAgsrqdAKl9b/miUbgHIMb6Ns9jw+BMSURC/tAOpayrKjrhqaZ8f57L/nJp58wugmrLeM4cP/4wKvXr9luN+x2O9n2TJ0IqW1GIDwRPVGgDT69NQZ5PifAU8Wn00O2bic3+FOBEjsZbRTawDjB6MN5u9h3Hf/+v/u3rD/5GT//5S9ZXV7xxRe/k5Rfozm0A8ocefbiJVXV0B5bSYVJ7pl916KAqsh4eBRCpI4K5x1lkVEWBZvDMbkJjCJnaerkBjpjtZjx2ccfcrmYczgcsFaz3a6Zpok8K2jmNavVM/a7Dd9//x1ffvnV+TT/5KOPyIuCSGS73QjLOl3BeZYnprdjvz/QdQPGtOx2O57f3rJYLji0e4ZxpFnMMZllNmsSETLSH4/sd3vKIpeFQVZQNzW73YYpeIzWjINDqYjVitVqKa6c+w3aKPqu5+3rtywXC4qiYHts+fu//zvW+4m+H9FKEnaqsqCdJk5cvRMdwKe0AJXIjcLNFJ6ct4ZxmoQ/p8XPPNMeqyOZlXy7g28J4STjOkEM0mVbBVWeMS8zZrmlygyFDZIOHCZhtZ9GS8QBoe06nJP7wWqIfkLnJYvFguXqEpsVZEXO8+cv+Icvv5Y498mDtkTABYkuU1qjFXgXmfYt7eiYVSWzWUVTZRwGTbXfURU5dVkKgVuBURIIKriT3IvOyz2ntQTuOgQq8QEmD8MkbH43ifwIJY/jf2hg+T+/SFmb5mpFAqGetnYJwUeJFkkl/ya0QoU/hJrP87gW2oLSJ+f69DVAxL2CZ6FOZnfhD7YiacBHDOyTYlzplNgqndtufeDx7rWkVYxiaKaM5ebmlovLKy4uLvmLf/YXLJYrirzksD9w9+Y1b99IOMN6vWboegE30ShlRWCthadidIbWYiGsjKQKcw7WSl1vwqZktSvPzphk1zuVxMD5VDJBs3l44N/9d/+OT//kZ/z5X/4lX331ex4eHpmc49i23N294+XLnOVqiY+BPMvp+p7Myhbr6MUxoO97ovccj0eJbS8k9sp7R997Mit8qcNhz9B39EYzdT0qBpbzGZNzPHt2xfHYcnv7nNXFJYfDnrdv7nj9+g1GGZ6/eMlqtUBrQ5blbDdbXITV6oIQIn3fo4zEb1d1Tdt21HWT8EXPNElwRFXVSZUv6+kheRnVdX2OGjvsduR5ztu3b9BawiiCcxitaCfhWVlraI97VAw0Vc31xSXHtuW7V6959fotWhtGH2i7kcOxk5RhKzmIgxsJQy8CdB8Yux6VOrUQw7kbcj4QtGL0nskLTpObjNxaCVC1mlIFBidY3jD0DONEclKW7Z8LZFbTFJZVk3O9qpg3GUWm0TFp3FKnraxGY4gTdMPIsZUEI8G6BJ7I85zlcsFqtaQqK7pBgm4zK2ndQ5hwaHw6UX2MEOS4nRS4oBn9SDdOdG6kGwuqQTOrckJUGJNTWEOK1kOSlSZCMq2UjnFiGkbGcWTyCcJxQjvouz6l2yQdrBZy8h9de/7ob8ySY9P51f4BpUBFghK/8VM3JDRPCRiMRmGCFJanpWa6nWOiEpx3q8lDWaVk5KhAPa0rQzyROjkD3TGc6AYyQkpahoxVxmpsNHivZRPhR7579Q3ffPc1KGndtbbUdc311TUfvPc+n/3kp/zzf/4vklVty3a74/Xr1zyu1+x3O9r2QD8OZFiyrMCYExv5tFaQC+HU5SttUEnaI68B2KzGZDXi3oDET8dAxBAifPnFFxwOWz759BOePX/OP/z2dwyjCGc3mx0RjTaGaRR8qe9b2XhZjZsk1t3HSFUWiQcDeWFpjz2zWcXxKFSGly+f46eJxzdvOXRHVouljKxEqrrEvsyZNUseNzu+/P1XfPfqe0zUvHzxgpvrG7ngpsBxd8TFgLIZSh+5uKoYpomL5So13gLAxhgoipKbm2tsntP2HcbJ5nRynmF0ZHlBWVTEIATAsixYrBaMg5A1hTAcqMoS7wb8OBA0KJsTfMCWmsPY0+73fPPqFV+/eg0qx+iczXEgBMUYAsZaERz3PVMQC5UsUWFcN0AmRonqzD2QEc8FGIF29OQ2UOaKQmnyvCDmhWyUo4cwEFzGXsnzCkE8DXKrWNQ58zpnXlnqLGIZ5UA/bwPlLhldZPSafT+x3h3px4mAEU8rLfdGM6+ZLeZkRcYUJgY3AYq6LKnLAo9MVpMTSMMYK0snTpl/AQeMNjBGx+BHGl/QT4FuhH4ILKpSipYLjKM4sHs34JzHpRSeaQp0vSyI3OSlcI2eafJobcmSK4cxBmNPvm0/YpESbc8pE8+lbYd0TOIldW5mZesDsv1TqfPILCD4lGT3CbnuJF48AdWkbZ/g1EooDfGklj5xN6QYhOiFZxVPE6gII7UysjaOJ8GNGJahDMZoUIGg5bFkVkyP3Tjx6tV3fPft1xAVRVayWq64vb3l+vqGDz/8iD//87+QU/2457vX33H35o7NZi1JL6dnkdgRp7JqTxfCWet1ojFIt0iytDBaJe6ZRgIBFNvthr/9uyMff/Ixf/7nfyFJxPf3jOPIZrNhtVqhiDg3pb8vb7ok1chpOXiPdxN1U5JlhtXFMgk7I+NkzmTI1eUFZS9bKqU1WZ6TFzlV3XA89jw+PqbNYcm8ari+viR4R101uNQFXd1c0zs52UFRlKWwm0PkBMSMo5y2xhhh7QOH7ZblUorjfLnAT07cIvMMN/XJIXIQoL9puLt7Q5nnHI4HitwwTRPzpsI7x7yp2Gw2PD48stlseHjY0PlIN/TgJrzO0MrgvEtR9Jd8+913AsTHmOxFUg5eenyn+EZQYgEcIkTpeo7aU2YBm0VMEIsgE6BWCl1k6FhgLQyDx00KGyN1ZlhUGWWmqUpLmRmMlq1fRID2zBb0Q2AKsG8d633Prp+YooxpmRWDw6oqubq+5Ob2iqLMiArm2YwXL55zd/+OY9uRHw603UjMFcaWtMNE9JLdJxtNjU8DUhgdo584pO68ygrWecayKriYV8yqgswiWkaThNVnT7aIm4KEjLjE70q+bDbZwGitJT4rkzH7xy1SJ8dKFfEkF0thcslIozQh4U8q6bL0iQTJyeDepE4syt9JAKSwMvQPEiVSv5XIZeLcmXrNNO8JaTPxo+BMs5dJ1JwlDDEVKaXt2bDLJLyB+LQEEF5HhrFSREKAbmj5/tW3fP/9t4nnYpnNZzx/8ZxPP/uUz37yKcZYHu/vefP2LXd39xyOR9loItayBmErW+S10gk4JHHNtBFwMjdpXIincVEeo/OR333+FTe3N3zwwUcsVhfc3z9gjGUYRopMtkxDcvA8bUjzXFbay+UCovhcF7M5eWYFR7jw9G1L13fEuqTMC6wxksUWgoQv5Dld19P3a2J0XF2vyAtNVVaUZc4wjhTZEmsMxazA+8hysSJqGdmmEBiHDV3Xsbq4RGlDVZVMk+d47FBGc3l5KSGpbUeWWZHHOM/9+pHlfC4j2ii2LV7BcezlKIqSzTgME3nq8sPkxKe9E8uV+8cN79ZHuqhpJy9gdAhkaelSNxKGqa1Gu4j2QQJtjcHmGVNUHIcBHzwmBlyMoBQuQRsajZ0iu34CE9GZpbYZBUkyk1u0KrAGXKkgae4KHclNpLCassyo6lxwKJVCPJRFBRhHxfbQ8243sT6OdF4TtCazlqauqGzk8mLFy/desrxYoawmOKEH+Gng5mJJGAey7ycOITJFjYuG49ARpzMV86yQQEm++OAnRidQTKcDvXXCFp8c4yKwaEqapqSaVWTWMrQtx8OBcXD4NGlprdCZQUtvwunoNkYnl4zsjPf9aEVKn0gRiPl6PFHfQSoxTza9QVm8S57l2shGLBm4nywizs5QZ3bqk2D46Uk9ESZ/qGE6Uyd/yBT/AQh38rghMaBVLpYaP7QdTmfWE9h9en5nq4r0xqXNn8AJnof1O+4f7/iPf/3fk+UF19c3fPLxJ/z0Zz/jn/zFX7Jeb3l7d8+rV98z9kcmP0p/mbo9+X2SDmj1RCjV6hwHRjrJQ4hJ6Gl4d7+m6yfe//AjPvn0TzgcDmw2a169foMmiF2usRRFTkxbmSIT5nRmNVlWYo0hS3FWQ9+Tzecsl0sgCLh94hslUPN4OJLlkmy7XMzoxp5h7LBGMWaG2XyGNgLOF0VFBNpOik8/HinrhizL+P3vf0/dzMiLgjdv3lLXzblTbtsOq4XN//j4yGw2oygK8jyj6zrGXoTEt9eXeO+4ePGMr7/6kuPxgMKzmNU4DbvdlirP2a0f8X6iKDM8hhHLGCLOaJHdxYhVYsBms4z9/oCKkGmDtpHMGJo85+r6kiEGHnY71vuDQAooCYmNiV4cNSYodO+IjBgj10thMqq6IVhzhjJiot8YIhkeS0Rrm7A0Sww9ENAGvJs4bgfWmyMP25H7w0Q7BZzKMDbj4mJFncFFk/Phhx/w4r0XzOYzJjeRZYZp7JnPKtw4ZzzsGOqC3Acetke2uy1ujBCztJ2WsdJHjfeILleZ8z0wBkOcApEJ56EfHe0wMh8nusnTVAXTMKRUHpHOZbZIFIW0QErOCd47dGp2xK77xw4HPe24pOA+uSAkXY+JUaYxLeptZS0uTsQgNg2y1Uvr+iQriQiW9UNgXcD1H/5U6bJOeS8CYMZz9yN/VGn7Es+PUew0NJkSUt6TBomzcPkHi0pOc9qpg/HJpYEkPPZeLFRdemF98IzTSNce+fabb/j3//6/Z75c8vHHn/DZZz/h5z/9KTFMPL57w9vvv+fw+EDftcLLQPynTOoSTRqJlUrdFLJ80FZGD21zsrLB2IzDvkObnOvrGxbLFUN75NuvvmC1XHB9fY0xlv1hT0hJLZMbURHquuLqciU/Jyn4pyQ6tZlsqYZpOqv7p8mxXq85Hg60xyPH9oCxhqurKw67I+2xIy8qIhpjDbvDgagMtijYH4/cvnjJOI588803XKwumEbH4dBS1zWH/YGqqpjGUYqltTRNnZwixc9q6HvyzFIVORerBffv7hiHnuN+S3CiQvDBcTgecOMI3nMce4zx1HnJft+ya0cmL726D6IdNSAAMoqiLIjTRGlyEfgqTVOW3F5dcn17RdaU3G82fPv6jldv17hhIipDMFo6Bi9Qh46RGCaMOmJRNMsKYxTOTeAcZeIUxRjQIZApRYbGWE2mA2oc0cmAbvKe9a7nzUPH435i03uOk3zN6sj1smaea6IbMaqgbhqxvCEmELsjxBHvR2Z1xfPbG7T3rPMteV7RzCZ2nWM7jOz7gd57eX2iCJ5P80pIWLPD4Yg4J3Hzxw660bE/Dsz2ws/SyPPCC2XIWJOoC2ICEGPAaJimU7MhAbnO/chWLadOQ3Fy1FMyXumkiFZRPpEiFtLWKkTOzFUVTt3YUxdmtMZrLbwpqT1/0E3JOHayk0haQUJKLbaoENDKCB0gxqQRPCFkQR6fMpzExzFpnk4jZSCK53SUwhSSC4NzLgVEiiFaTGTOMxiuzJknZqwmOsdhu+Hv/uZv+PJ3X1CVFS9ePOP9917y81/+KU2Rs99tefPmNffv3rHfren7Xix+HUQTyZQgVhAobcFsNsPkOUrn2CzH2oyoFMPoeVzvyfOcP/npz1Ex8vWXX9EeBz78+EOGwRHCkKQbyW/aarpObGKt0WRFQVZkTwU7COl06Aeil0JhjabIMqgrnFuw2+1Yr7dsNns++uhj6nrBODq++OK3hKB49uIFZTNjGB373Z7FcklV1WhjRRiNom5kc/jNb3/LNE18+OEH5PM5u90+XWcCrC5mM6oiQyWPqHHoCJMjaJg1FYMOdO2UYtUUk3NkNlJVJe8eN7y+W7NvRxarS9rNmpMJpIoRZSRA4uHuLc8urok2Syz5wKKpuLpY8vz2msvbK16Oz7m6uuHlix3/7q9+zbYf8V623IJPBXo0MWjiYSS4PT7AbKrI85wyt8mOV/Rr0UfJB9BaXBqUEnteH8RT6nHPq/WB+1ZxCBnTKUFIeW6WM56vasb9niwr+OTjj/jk44/J8hzvArnNyGaGEBzHGLGVos5LpmGi7ycG5zB5Tl6OFIOmbBXbbqBzkfHEWYo6LavkLhJ4J6K0ZghacOQhoLTIvSZnKDJDaTSZFmz1lC4j8Vdyo/s06UQVzzK5kwPoj1ekNALsnkYxnbRvZ0zqxKDlrHEyWoIWfQxYbc6xUX+gfk4YCsl36rQJSl9MiFV6AGkneDK0MzrpoCJofZKxnHqhRANI8UsxSsEKIRKRouOT/MVNouXziR8TYjwDvJykOurJy0olLO70SHUq2CrKxeenkdY5vvh8z9dffEFRFMybhvffe8nLly/4+Cc/IXjP/bt3vHv3hv12S+9GnJdCldkcj+Y4TMwy8e4uyoosK0hLUoaxl+QRFfjwkz/BB8Xf/M2v+e71G37yk0+xNmc2W9J2e3YP92z3G2IM3FzfkOU5KnFZTq90kRdYbSjznPZ4oDu2RO8pComGVwq0NhwOLWVRkmUZXdczugltM957/j5lU9EPE/P5grqZoZTi/Q8+4PHhke3DI598+hlN0zAMA7e3t4zjQJblVFUl3k59h8kzyjwXDtfY892335JnlllTyyKgbXn9/R3b9QOXF0uyJOXJrIU4cffmNYdBuqabm2d89eadANGZpPBaBXWRE40lTI6yyJmiZ7VYME0915dzPnh5w82za+rFjIVeMV+smL2+YxhHvvj2NffrHaMLSeUPvQs4rRiD+N6345qLYaCpc8rEM7JpA5yloFVrMvm7XU/bDuwPLdvDkW07cRgjvU/5fiFileZmueDl1QX9YUueZ3zy8Uf86a9+yfX1FUWZM02j2P2oiNFQFiV5ZpiM4eb5DXVds15vePXqFUo7QoS+myiUBysH7uRPkLE+04xOnYNsNSXhaHSBYXIib0sQgjIZeaYwPIn548k22VjxFIgQPUlOFHH+RwbOdaIFoECFp6gouW8VIVEGzq2UXNroNKqpVLAI6WvnMYvUHcktf3pdIIk6E7AXE+dIsrtSZ0SEk68NEBNVQYqVTrYt8khEeJzCH8Iffp6FyTEk7xuxeDlZBaenLb9PBUopSWpWaVPpfRA30gjWZAStybM8WbB4Ntst2+2Wv//NP9DMSp4/v+W991/yq3/yZ2gUm/Ujm/WG/WEvHZ8xBOSCmCZPXSej/ZSPlk8l/dDTtR3r3Y6Lmxf89BeR//v/7f/Ku8c1/5t/9S8ZXOCvf/13vHr1rQDWbqLre6qiYF7XVGWFArquZRzFvdEYI0EMxtLMZgxdR2YcZV6wp+Xm5jYtPBSHQ8I4XJB/t2momxkXV1dgLG3XkalAM5/TzBfSUWgBy3//+9+RZzlumnjz6hV1WVJklvawp++OuHEA71jMZ2zWD+AyYvLPLlJQbXCeQ9tS5BnOe9zUM4wDh12LiobXr17TjU74delNLIpCQiq0YbvZcn//hkwrykLG76JQhHigKK+Zz0ryesElimZWc315we31N/z6b3/L67cPtMOI1+I3PkXF6Ezykwr07kh16CjzgqocyHPZ8yotDHZjLC5EdrsDx2NHNwjGMwbFFDQOTYweawwfvXyfm9WM2B+o85wPPnqfX/3il1xfX7NYzCVOynvG0eH9KNxBPL0WVrvNLc2sJMQZcM16vSYE8Z3KTSB4mDR4/4Q5q9NEkjCaEwSiVcBFGJ3CGs3oIDORYCEYubcVWpZSSuNDEDpCkPQZ75OAnyf45ccrUuZk+sVZ9gKc/ZyIJ/BbcTJ4PwHYJ5GxJK/odPPLiyBfE/KCPHh19nQ6yQjOVSJtvk7sDjHMU0kn+BRdJfiTP3tLgWBIIrnxf+hddfK14imMQQDk02NMb9ypw4uak5O+jI4pwcN7YZUjjHurnzo/sd1NhU3DOIx88bvf8v03vyfPDdfXN3z08af85Kc/wWjDbn9kvdnipvGsID8X5fRhrKHSFVpLMfi7X/+a7779mmcv3uPm5pLt/sAwTeRZxV/9x19T1aVgQF3PTz79FOdkvNBRmtiiKOnbI8SB4JOjpPd4Lxa04ziy3RyoZ3NCiLy7exBtmtasVpfM6pnIRLKMvCx59yA2vjFGiqLg5FU0TiOZsbz38j2maeLh4Z7jYc/i/Q/Y7/en44bVck4/TmSZFfyqaxmORxbzWYrL4qwGOOw3zOcN2kdm8xnKFnz/5jumKeGHUUYWY2T9nec5l4sVY9didKSucupCE+OEZqCpVmgCVVljixILPLu5ZlZVzKqa24sr/vu/+lv+4Ysv2E8uHaaGCemsnIeud+wnyHqHOnjhwmnONBGA4MUVIATZ4k5BE8/XkGK1nPPe8+dkBNb3d3zw4oYPX37KJ599wvPnzymKPEEUT/eedyc4RMJ1o5sYhiP9seXx8ZGH+3ccjy3Rj5RlxuADfozoyctWkae91R/u3k7ysCcm/uQD1nv6UfzmcJayMKIOUSqRXcWHfXRiCOjDye5J/8Fr8f/r448vUsBZMHvaVP0g+ECT8KAoWw+VlOshAXIxKvwPPiNi5EU88ZoUQZ9w+FNHJLN/jCoVG5U6JPmpzsnPF+5FRZ6ExzEqTKYpylKSWZWsrJU+DY+nooQw1H1gHEZJc/EyK9tMtl1j39O2LX03nD2WpmQnE7w/Y2VKI5wrI59RaVwUwzJj7BMWF2Vjl9mMGDxjN/L6u1e8eX1HPVtwfX3L9e0Nl1dXlJkITMdpSoQ8D1FY2jrRJ7I8cnFxwWeffUbbHsky6Vbu14/8/a9/TVPXfPjBZ3z9zZdolWN1TnscqIsZZMJEny+WwkC/8Lx9+walJ4rcYntD3/WEqGn7iXq2ZHKe/fFI14vL5sXFCucC682eogm8f/ucwUnskjUWm5XE4MlyS1EW7HY7yqoiD45h7Djst7x49pzvvvuGqq64ubmmPx7p2iOowH5/YOo73n73GrzjuNtyfXVJf9wniY8H5XF+YAqBsix4e7dmuVzySXHF56/eMPRHObSiwrmB2eyaWV1TZjl1leGGI2EyXM0rFoXGaNEKaqWoiorRi6NGVRVcXswEqyss82XDN6/v+P7ungnNOE1SeKJGYZkc9CcOnRK4JJ4WPJGz75KkZ8pGSilNU1U8v7lmNZvhxp6hO3J1ueRP/uQT3nv+jLqpk/e4psgLjBJiZ24M41AyDB3ODwzDiHcTmbU4awEhOBdFhjKWaAsCln46opX4qgd1emgnr6hTBTh5t8lWPJBsVzToENA+YqLH6BxlDQL9KlyQa8GHU6SVOm86+cGh+6MUqXPi7dmaNz6NrD/4WaeO5GQ0f7YLTcJiGbf8uSsI6DQ96B/8rCCjVMJ/rM3JMwGPsxRAkNkESGrRFDo3MU0Tw9iLyYAWrdM4DYlP5dK49/RgQwA3inDyBOxpbViu5iyXK8qypMhyyqKgKCpAnTszgHEcmKaJceqZppGu7dmsdxzbIyAUAlm9Sgt8esNdCGcNEyR9oo8cjx1aP+BcoD10rFYrZk2DzTIikck74gh5DsZkKKPQUWEzw7PntzTNf8Z2s+H+4YGubVksFnz37fdcLBf8+T/9C+p6hjI5bTdRz5csFiuKFFwqrqqwvLxEAePY0/YTLloety37dmJ/7OiGEaMt/eQpm5x+Chy7juvrGUXTkGVC4FTK0B57iqxgs9mgrVj9LpZL1us1bhxQSnFxsUqva2C1WjENI847NpsHmqYWykSMVFVBYWpWywXH4x6ix7sR58Rve7vd0R5GLq+XLJYLfJx43B+wKLQotBCfbomCV6MTO91xpKlKFrOaIlOsFjOUikzTyH6/o15cYrOMLHomN1FUBdd5zsXVBc/ee85X37zir3/zW7599Zbd4SByn2HEB06jgXQlETF/SwdV2pWjlcFYRaYNZVWyXMypq5LgJjYPdxgduL5Y8ulnH3Dz7IpqXlHmOUWRk+cWa1MCksrJMkNV5jhXMrmBh/uR7rAHpCNuZg2ZtUyj49iNuIcNVo0YFFYpnEoW24pzPuBTIVGnNdb5zz5EiTdT4JQUIu8DwaQsBC2W1NpI0lRUIpiWpCdxtfhjPv4RndTJVU9W6Kf/SxITS1JLfHLGDOHJ5ynZ9v4BDhSF5IkS3oTJLKUtyDJLbuXUzfOcED1ukvinaXL03ZHD+kjf94LJdB3D0BOcAxVSBU8AeiqUIuj8wcH1g695H5NfuFixkq6tE+1KaxHJZlmRbGQUZVFSNzWFzdIKuGA+n7NcXvDs+UuWyxlZlhGCPwtlowtJYNvS90f67sAwdAx9d+7I8rygaRY0TUNV1gBC2CwLUMnDPIptjGNKOJyIULMsY5ky6C4uL6nrmr7tOByPaCUJIB747tVbNpsNbx4e+Rf/q3/BcrGkKC15XqFVSec9h/1BKANT5N22ZT9Efv/tG/Kq4uLykr4foHCsjy3heORidUFWzwja8OrNW6q6YpokOqrteom2Kgog0h47jDG0rfhL3b17x7ye8+z2lr5tKYuc/W6DJvLu9Ws+/OA9VAzM6prN+oHvvv+Gxaxmt75n7HcMfY/WhuNhYLPtyKsCbSL3D28YOqi0pdcGN04UWSEeZUOHDYFlXVFmc8IkuFYMI+MwMG9KyjwnSyZvs9mMqimxmZBdu+ORIitYLSp+9Ys/4b0P3ufXv/57/uHzz1lv1vSZSl3VKQJNOo8TnKFINzCaoiipihybpEuEkeO2pcpzLhY1z26v+OTjD/jk0/epS+G9WZMJITKldFuriQYO+6M4nE4jm80Djw+PKDxZJgWnbzvevHnD8dASgyZgyZQ4dGZAMIkLlmAQZaSUhhDPgQ1K6ROlWrqpc4eUFB7n++wkb1NnD3WrnmCT4CH+2J1UTAkwp9ThE8P7zPSO0kH5ZCh3wnZCfEqSOFmHFkWJthk+BrTJKMoatNicTH3PYb/n4fGBrpNQw3Ea8M4xuTGt1WF00gGdCqZOBQVOnCkxzD+R0s6gPPJXTv+OVmJsD5CdGtr45LwIAh5Ozp2xpYM7CmkROPlQnd5EYzVZJrFRVV2xaCQ5pa4a5osFz1+8pCoLbKbFVjh6+q7jcDgw9gOgKIuMqpJwA6WfsAA3TfSdSzQCcbaMijMRVFlNZUrKosR+ZGnKirzI2GzeMJ/PGIbAb37zO7797nt23cjtex9weXlksZhzfX1Jlhmyak4ZLd16I+GT+56vvr/jm9d3vPfBB9hePIV23UhZFDjvqeZL2tFxGPcsl0uyIOnG7XrDrGnIUyS6RIGpZA8jFI+qrJgvFuRFzn7X8XjcczwcuFwuub68oO86uuOe3XaHH0cW8xl5eo036weyoiZO8O03dyhT8e5+x3KVs5zXtMeJtvNYpci0Yjmb0XUDKiqmaaTV8Kuf/xPwPfdvvuH2ZkVe2jTGXrC6vCJGQ9/2ZFXG5dUVw9DzdhogBmZNhY+Guq5o/tk/4dOPXvL69Ru+/uZrHh8fORyk2HsvY57SFqUl6MCmwAOjlVB4/IRVIpyuqjnzecOH77/Pey+fUzc5KjqcC9R1Q57lZ3H4qQkIMXDc7/CTjHh93woOBkTnsEYkNM9urnnHI+v7Lc55TIBVXVPk/skz7dQvGSmu4+QFolGaEE842kn4lb5XCXdsTGqLk1WQ9164kerEdzxtyiOnfL4fr0iRRoLgk60KT7KUxMdxyZ1RYs09UxDgUm5eS1mV2ExwI+8DbdfTDyPrzZbdfs/QSRJrOEVdRRnTTvYsEAlhSsEKHq3DGTA8rQSVEsxK0o5PBnoCuGutUcYkz+WIjx6lNdbY5BqqnrhQJ9pIWgDEKE4QNhl1aZ1CIDh1a6fvS/LpGAlesd21bHdH4W0lKUWWWWxmyAtLnmcsmhnXN9dcXt3IqV0UaVHxxBsb+p4xjcRd1zFGyHMZA7XWVGWZ2muNmwQHqZqan/7s5xyPL3jcPLD+9jVv3t1jsoLr2xeYrAZTsD106PxIU1fC3E9YkrIF3eB497Dh2I48rvcculHY5dEyRc04OXbHkcjIfD4HlbE/9BTWUhQ1ZVWx3W6xRh7bd9+95vbmhraT3LWb66tkIhjI8pJZ01AVBVWZMbRH3rz6HhWF/zQSeXx4REcHTDTzJZNTfP92zc6B8p780KNtwI8BHRTRO6HDWEuuc45Dy344UhWWrj3w6vVX/Iv/9M/QcQ8GyqZmNp8JETEEHA5G6KZOtoNlwXJ5kTp42aTVeUF9c8Gs0FzOSj774Dlv3r7l9198yXazY3844ELCUJWckrnRWCuWykVuMQpub69ZrRY0s4rFfMa8acTBopSuvC5LmqrG2kI2ZcKuEaeBvqVt97hpwKhI08jIeNzuGIeO43HPdruFEPHjhDWG3BYondHESD8OBJDk6xPo7T2jD/TWM0UtIuWEDTvnpBYgHZQnMoQATriRuTHSpWm5F0nFVHDZpLF9Qnh+nCIlHYv8EMWTQ6DwjQKnJGCfzOuikps6y3KMtbggY9vxeKRtW4ZhoO8HRueYvBNtlgucyTvxZIwXzoVQxkSeOpzIH/wZSK6FChB9lrgIZpgsk8eT51hbJDAzSmx6lpMl36OT5alS4lzJqXhFUoKtld8bQwxBSI/WpgwzJfyqYcB7CYa0NufUHIupnpwiIUxn99Lj4cB2u6HIMxbzBRerJZdXF8wXC7LMyM9Lj0elbL9pGOi64zkooZ7VLJcrbJZBKmBN09D2HZvdlq4PXFxe82f/5J+y3e7Y7PZ88+13/LNn/wnX15eUZY5zk5Am/ZnfT14WbLd7iqJEoTjsWh6nPePkzsTM0SkuViuUzhjGtzy7uSaipIDtjhyOR26vr3h3d8disUAbwzCInfFsNufyYiWPl0iYRoyCcWhpu448LyjzjMfHB3abNbvtjlldcHN9wfHY0k8CTm+6Hms17zcL5osKNWru37wmyy1ucCit8KMTW5U8w/uevMyYzTLa4wPPn18JlqgFW5HI+Y6szAkEyqoUh88QWMwXFEXObrunbXvc1GOMYVbnZGrGYa94+ae/5LMPPuDVm9dsd3tccGw2O9p+IMsyVvMFxhjKqiL6QF3mvHzvuVjfGIVLWYPjOAoPLM+IITJ0LabW4q+lJfa9yC1aeYyqGDrPt998Q9+2TOMomlsi9/f3tG2H0SLIngbpl4rKUpU5Nul987wgRmi7iJsGonO40XEcA4MLyTkhNQXRM0lnQEza25Dsw3Mt972OAWMDg3NEPz1NMJG0MPgRi1Q8F6K04g+IA2XCobyXhFeVZWS5cHm8dzKy9T2jc7jTZ2r9fcq/89OEcxOn7DyRMQit4FSczuzUKBs+wZLkRgpRfnaW5eSZ2A2XTc2sWTCfzSgriSjSWidtnk30fUue5+dZuygKTr7VOunq0jSbCo7FmGSbmr5Oupl1Ek5qY6QIIeRUa0RMrFPtNcrKmIecLFqLz/WURghZRESMleSSk9RHG4ONgYKc4Ca6w8huvWG/23HY75jNZuyWKy4uL8lycTIoiooXL16wXK344OOPxLHAefKiwCQTtLKo0Cki3gfH1cUFh8OB/eFw0n+TZxm7TqxgxnGkPQ5M3jNOTmgHOgNlsFnObF6z3e3oMsN6vaapxJal6wbKqkm5ezl5UfLyvfe5f1xTVCXGGvFdmgb645Hcavp+4O3dO64uVnz55VfMqorlYsnY9zw+bBgnhyka8rQFNSbyuF6T6QmVAhHc0KKMwnuH0ZFFVVIUWcIcJ9zUs9tuqeuauqmZponH9RofNGU9I4sFwcHYTWRFyXA8EP1EUZSsFgu0imzWa6YYqcuC5byhKnL6vudiNaMsP6Afe1BROFyHA9M0oaJCW8tqtcJPXrgLYaLrdtR1RZ4LRlXkOdMwkWnDNEzs1mse9T1FIfdY8CIUf1w/MA5HikwR/YCfRDNa5DnDOGHzEj06xtEzTFFe6+Do3EQxWuaLJrlXRIZ+Yhz7c6DKOE5Mg2d0gaBkKiHxCRUwOc1kdWL+wzRGBusotBYGgEEyEU42TycG049NQTgBzSGtFqPSiSSn0TrDmAzCxDD0KV9rSL+OZ1xKiF2yvj9FKYUgiRTO+dSZ/RD4Pt2k+nxTxeQzXdU5ZVExny+FRFjXLBZLCZHMc6Hx95KAstvuuLt/kA4nyjh6Kkynjd1pxAzJGN9amzq6ZCivTfr1NIapM9nzBAaK7/qpoHvxmUqGdqeVq1FKzO+SvYWkt8vPyIymaWqaqqEqC/K8QJ8ZygZtNFWRU2YZi7pmNZvz5tVr2t2Osetp9YEyLwhVzv545HhsWV5d8uKD91llK5wXQl0MYudiregap2lI1A0vBnpFzmK5ZBxG8jyXyHI/Eb2kA2mtyFV25tNs149sdlsOhx3Pnt8SLi/o+5Z3795xuVzSdT03tzdM40A+TFRVRV5UzGdNekyezW5Pbgz3d3e0ux2LpuLN3Tvu3rxNa/SM/W7PlI+UZcHhcBQirY3sdhuauqTrB4bBkGUXdG3LoT2mm1FcHbSCyirG456X711Tlpq6kKWES7HoxliszdnvtxyP26QI12SqJOrI2PX03ZGqaaiqmnlTgZs47ve4UThgRWZxk2zy5osa08pypi4sy1nFlHzBlFI0hSGbNWcTQBUlBbksS7ARgiIzBhWikG1nYhcdvKQAT8PIfrNhd/+WoT8wn1fM64YmK+h7x36/Z78/sjsc2R+OhJgwpoSjGqskvh2RsghVyEsqlE5NSMKVVGpIxPfyJC1LmQVReJCjdwyjo1OaDIVRElxKMrvLT1PHH2h2f6QiJbfYU7E63+RKcJlpGplG2VaNoxQroR8IU3VyP1TYA2jCSYoSdPp82sqdAkitFSCzqhqa2YzVckVV19K+ezHU6kfhEn336rWMke0gqTLDIL7kgM2kC7J5llxG1R9sLGQLoRPpTkinWZ6f7U9U8hmSjyCdVEyM81Pben7VT8ESqdMSEy55vF40T8FPxJiKo6w60EhkV1XWPLu95erqiizLcMqIbavRuDAxjRMaxfGw5/7hHfvdToh8k+N42MtrqMCjiFoTjSKvKpq6YbUSaxQVZRlwPLRsNhtCDDRNzXqzRiHWL1VVcXN1zXK5ZL2RIFGbmYQ7PhFdvQm048S7d+84dge+e5UxTY4PP/iAfprkEKlq3m63dF1P0zQ8rrdoY7i4vCbPNMfjge9fvaKwhu1ux/fffk1mFHXd4F3AGItWBqstwQlLOssKykJ8v1BHYhhYLhZUdYmOCh46IorgotyMMTJvKlwG3WHDxeqG6B2bxw3Pnj9nu9kxXyxQeMqi4u3bN7wwkBc1MXrcMHBoOyYf6NoDi+WSuq5ZLudkVrPf7ej7I1obQnT040BVlsxndSpAcpGoGqqiSNbMlixT6CInhAyTS8Cm805oE1rjQmQYWrSJgtX1gdevX7Hb7Yje83B/z3Z9T5kb3FCwRdEPnmHw9N1IP064iCQqJ4wzKk1RFTSzmrKwVGWWHGcN4+DScupkdXTCe588y84THymwzQvPwmgBhbwKZFphE/VDwlVlG2h9+IEA+UcsUqc2z5gMrU+Bm5OQtdx0Nvr34cT4TmtJxOs4BJ3gJo1WIYGZEYJn8lJlMyNBD2VRMJvPWMyF82KtZRxGtvs97x4e2X/5FeM0SpHUwlCX9N9BUmd5KiplmaO0tNZZJikX2shYplFPTPBTETq7K5gUxqATIC0mY090LiGpnTV8aXtxwtMisnk8hT8SIlpNRCNx45PSqThNIsZWwuq3xiSAtCRPMhJ5TIJrWStFduwHbGZYLGY8PuRsNxuqsuD+bs0333zLfDnnk5/9lLopyGwGIXD/7p7gI4vFUtxClcIWlsvra7r+IB7iaeRVSnGc9hJMasRxNTqxtVVhwkRFCv5Ga6itIQwT/eGAcznXz56LmNVKV/rm7Vu0zYRKsNtjM3HRHKeJaQx8/dW3eDdCU1KUJYctXCwv6I97pr7HT56u62iqmnGaaLvAzEricp5nLBcNVudYFZmmjvvNA0NIXlDBkytLXRSMbpRwkCDQw+XFBXdvXzNOHmNytLLc3z8yny9Aax4ft6xW0PeDdMxKgTbsd1sIE4Zr7GxOWRTYiyX7w572eJRFgDFs148pLKKmLIp0aAEhUBVFogZIUIbNM6o4Q2vDYjZn7Ef6QZwplIH9fs133z5yPLR888XX3L15Q5FlFGXJfDZj1hTEMOAmj1EZVVVibIXb7XHjRIyBYZCUa5tpqpROU1cldVVgi5xMZwyDdFTDOJEZuQ9MgMIKGC73W2AcnXRWZyK3YJku4b1BRZxSGCSQxSJFSgCdQMx+5CL1/1W0fMC7gPcBF06eA0kCQyAoi1OeGOREDwlUQ3E2ezNKWLKLlaUqy1TVc6IP9EPH8Xjgq6++pm2P9P0gvk5aM5vNyYsc5z1d37E/7ESWYuSkNWm9Kx2RFRA8s9JNWSuYlBEJgFYnMpwRpwYpOfLnVIBO/Kj/yQ8lRAfBciQI8vS3bCZe3sSYUnLl5rZakSW3gaLIya3kosUQGMYhibgDGCOv+eSAKEk5VcXl1RXNbIZ3E/vtlu1mLZs+NGPy+alrEXhn2nDY7qiLUtbhSuES1laWlQD//UCMkdxmHHdCTLXaUOQVj5tHpklcEjJtUFjQAastRWYwMaMbB5gcx8dH/DDw7Nk1//Dbf2DsR25ub4hXV7hp4vLykqjg89/9jllVA5r7+wf6rsAqJa4PWcZ2tyd6T3s8UlRS4Nw4oXXO8dgTNZRVjh8n6txwfTFj0eS0TcFuO1IXFqMK8iLHEiTSXkkasPeefuiTi4CnLKvEa1OEoJjP5qwfd+x3B7SGsq7RRrSYwzDgp4Hddsf17TOKohCTOw15IZQL8ekXGKHvOoZBOquh79AKyqKg7zr8NEHw3L19AGUEX9wW7LZ77u8faI8tzg1k1nBxsRJXzclRlTVlnqONZegd0zgCKTqKnBCNBLf243miKLInIqgxmtwYqlz8u4iBvmsZRpdA9IyqqhgDBBXwSuNP5GtElI+TEfDEUVQRxOIFVFB4ApYoPCwSNQEhcZ8DDP8nPv7oIuVDwqMSR8I7L6b0QUE0RJLLgY5EDTq32BAla00HofGjZNtmLLN6RtPUFEXJMPTstmtev3pL13WSfkxIhm+gtKWe5WfrUWuMjI59J+vpovwDxrgxUoS0NhhtJRnDWpGMpDZT0ogTqH1qjxRnXOw0u0mxkhdUOFanBtijtHRkmRGP9LwoyfOMohT2r7Wa3GYYo7HaiBFfFA5Z8D690Z6hl6zA47Hloe2Y+g5FoKoqbJahtBJiawjotP0U/pcU2b7r2W43bDaPOOfY77dcZBc8v72hKUumrqPd72iPPY/bDV98/jmffvoZz1+8YLWYC1E0eAk/qGrGcWCz3yRLXUWVl3LiD+O5cOYWcg05kAVFpiK5UjQ2YwwehhEVFV88/pZmNafIK9puwO727HY7Dl2H1fDu3R1VUbKYL1Ba03UjfhqZVRXr3YGymnE4bAlaY8sSFzui8zxsNtT1DG0CsypnNWsk8mxqKZTmk5e3DDsHrqcpSpwP6DBR2BqUCJmb6ob20KaFjk9jv8HaXKx56walFcPQsd1tGZ3n8vKS7faRd3f3aGNoZgseHx/OS6AXz19wfX1FU9ZS4JVmCI6uawVmEOo53SCpzVVVCxwyTcyaGUM/SnFzTgz/jjvaw4HMZuwOLd2hp6xE7uVzh9GWaQy4cUJpT1EY4SuNPcPkCdFjlCYvc4osJy8LyqoQzIuAyWzaIIsbiMjABrrOybiXllJKBcLkxM+cJI5RkcxA0DppcE+Hdbp3ZN2fGhct5gppERRjxP/YjPNpGgXodiHhSMmLSQmXKGqTWOQQtfjnCKtWtGtFLiTDk+1t27bcr/eMwz191+LGPvmZK5TNMFoIYGnaQZs0dp1ytZQwbYuywDjxy1apAzLmFLsjo51KjPYTVcCm8Y00okkHJS7rIeFGImVRgNhT5HlOXVVkuSUvMpp5Q5EsS4w2+JCM56dRoofaXqQ6wyjxRz4xz4NsZHyYiMGjlMRYT0MvhctHrIGqEL6KHgdOUdbWpI6zrhFr5BOGF+iHXiKuM8vyYkXX99zdvSVqRW5z1us12+2e2+fPef7iBRAYEuGvbmp5DgkzHEdJe67LirqqqKqSpqnJ85xj39FPTk5KC4XWgjEQxPhQgTeayXnG8UgIPbutnO4P9+9YrlaMo2O1WlHklvt3D5RVwXq7RcdIcCM6BPxQYAhMQ8f9/QPOj6AsbvRYm1E2tSwU0Ex9z6wu0ZmiqgrqUpwcqlyxmpc0sxkhwma7Z+o7QDBC5yacCzRNQ11XsiwBiqrg2B3h/p7b2xv6vmO73fHVV19TNw2//NWvpDhME0VV8fK9F3Rdx/27d7x+84rvv/9eTPwysUBZrWSR8Pbtw/l5l0XBMAyM44bFfMbYt3jvmDU1Ns/Is4xjboi+p6kz2kPP3au3aC2RX0QnYRuTpz92ZEYY6JXNKaoMtE4b9JNZZGJ7K01uzTnmLKggm/dRmOrTOBCD4KTBeWISmxdJawruDOOEqIjaEom4IDIZSSFP0pfTevjMFUpeVakn+NFdECSBRRJG/Cm5F9mcSptUYDTkpoCoyYsSm+coJSNg3/cc24FxPND3HeMw4cMkq4HExj37DqikGBc0GW20tKdKpxtcSHHGWMpc40zS5GlxKNAJZNVaElXE9yZ1VFqIpdbYJ/M+xflXXRTUxqSiVDNrZtRVhbWZeAV1R7bbDXd3b+i6jrbtzpsZokQk2eShbYz4pVubnR0hSOOgSokZKnoplIjNagwOghdpAsLjsjrHKTGsKzKLSiymcZw47A9A5MMP3ieEQHdsuX97x93dHZuNgNN+8rx9cyenohGvoLKqGYaB1XJF33dkRYExJhmwuXQ4BJqmoqoLYclbA1qIfiEGcgSfUXiJCyditWxjC6XwGrS2DCEyBM80tOzejWJB0x3ZDx3jNKEziw+e3GbURcayKHBdh4qBaRyJFORZxW7XM69yssRZU4xcX15Q14YYR5q65uJyBsrRHfdcXy7J91OiAyzZr8W6eJokOqtvO1CR7Xbk9tlzshTuoLR0Ht5PfP/9d6l7t+R5gQ+RV69eUxYl88Wcbhh49f13XFxccHEpW8XtZsvQd7RdoGs7jvsjs9mMxbwhuJEhet7dvT1TOjJrKMsSYzWvXn3HNE4QI+PQE/zEfD5nfn3N9cU1X3/xFbv1I0WZUS8W5GXDvBY/rugnvBuZBkdWSFy7MWVa8IiTRgigbUYMjsNxSDCGSosg2W4T0xJJCcXAKk2VaazWTEGaEaUkqGJyEu11mvV0OvDFEUSKok4eV0alK/2M+/5xH388BSGcgjqlfQteNCUSCZVRZEU6AXK8j4zDRN8PONenVlgIRxpFZjImxLlPKAkjwzAkOxeJxBJSpaRSoLJEvY54JxSGqKTTMVoIm3JI2LTqNGfAW/4dfe6ozp+pm8rznLqRYlRUJdqIB844DByPLW/fvuFwPNIej3Iypawx5504LcJpEEeFmITPEmdtbY7WVt7UJDGwKR5endYj6TmGtExQyZtdacFlqqqkyDIyM0u+Tx1uErZ/1/Y4L7ynU4JMkRfcPntGPZ+xWC7J8oz20DFrZsJ5KnJUiBwOOylYizlhmti1HRhxi4QnT3trDcE7DoctKkpUltaWyQd65xnsRE5A+yDEvUzY0+IJDkaL68VkLa33HIeBMUaGQTFOg9ibTMJ2jiXYIqMdJ+Io21FjLNrk0nUGj0VGS5sbyjJD6zEVHU+WF0mS1FDc1ByPI1rtuB+P+PFIncPzmxnj5Hhzt5GudD5jt9/y9u4tRVVKZNbkGEYRjO92WxaLFS9ePMdkmRj/7Q+8fvWaPM+5uLxkuVxK3tw4cTwcWC2XVHXFu4d3gHicv337muAjs0VDXTVorWiPIq1q255393dYI7Qe77wU5zRqvXl1xzhMNFVD3VSEGOjalnFwaOUkzCM3tIcOHyLd4DCdeGDVTcNsNjuL1L0X0bvzPl2j2RmCmfxElhV07YhzHT4I/hhdgCiavUyfNHka4xHcNcFLkn4k08nJIlylScikBkruWc0fXaH+MUWKaNMqXQYjpWMCovO00jeEEOj7MXVb4bwlOklOgvf4GHDTKDyqYaAf+sSt6gEwRokOD0VmTYrNUUzW4AiM4ygbPIVooDRoLbiNMoI9GZOlaq7Pujyb5eRZQVVVsgmZzTDWnD3Lu67j7t0dXdczuYlxkIuEKDQJYyUpmPS8DJoYRIDqghDjgk++6sYm36KSPCvldQiS/0cUjEordeaZGSVMeGM01ijKPOfyYkVd5WfezPrhXdpcCni+Xm/Z7/YUVXUWMpd5yepiRXASPW6sZRjEJUBEvxMomM0bWTJkGW4Un/MyTyGn+nTKiR84izm3t9e8ePaMzWaXIp8yone0zlM4yGzExICNLp2yEJVEzOsgWF6mtdiJZIYxwqQUdWZwKCl4iS+13WxpUVwXlYD9RqFwuKGlNHIoNXVOmStms4K8SMEZWk5xHyKFtuS5ZZw82njyQlHVGcosmaaBuppTlT2FzVjOlyyXC479kck7jscDlxeXtH3HerMGFO/evRMg3FhWFxfyXmnxFeralqvLS4pMcCzvHEUu4QRuGGgPezJlmNc1282Oh7d3cHUtryOK2awR/lCecTzuhWaRW9wwySYuiqDY5oZDPwKKfoqMUXO/PWIPnbzewVNYey48WW7wAbpuSKETQmFRSjr7pizRWoTlRgv0MU5Wsg7bATd1uKkXr7fTfa/0D2LoIj4FRxhAGYVLflmiTRSe4QmeObuoJFjmj5z0/nFFSqkMpYQUl+XC1pYwAifYRvJxcs6hk1YPdJr73dkzfByHszRmnGT71A8dfS+AuRQoKUBNXdE0FQOByQ0oSMVD7CGMNpAJGU/bHGWeWtXMSldX1zWz+fz8+IwxROe5v7vHJcF0JCY9oLiPZlY4I+FkmjeJ03RMis3gR3x05/Y4+JSG4URA3SEdU5ENFMVElmXoTGKmiqykriuKqkrC44IqjQ5FWeC9sIrX9+949+6t+Jr6kdxoEZXuBEjdbnaM08Q3334LSvHZZ59x/eKKpqoSZcKk032kqsTPPCIGdKeFQt/37Lc7ur4jy3KU0bIAyPOkz5RcQkVkGkVrF5y4YAQtuNN+ClRG0eiQRj7RDZ5D5pNbqux+oTaaUim80jil8UozTopDnNiHiSF4+uA4pFHeh4DGMZsVLCtLbiIXFzUGR55H5vMysesLZrMV/SCd+epiSZ5naBXJcy1LgaZgd9ijh4y6qtjv9yhjuX1xS1WWxL4V0nAQL3abb9Da8Ox2xdu3b3l4XNO8u09ERIE8jseWaZSpoUvRTjFGyrJksZgl+YdYS19ervDeM/QD7+7fsVmvWSxnAMznC26urmnblqKqeXbznNdv7vj8d7+n7yVOCm3IslKE9TGmg5jzPTPlUKlC0rS9UAh86GF7wFpDnluKomQxn1MWIpuyNkMhU0ZeiPqi7zry3FI6I8TP5PN2UlqgNCIW1gQf6SeJqjrRlDSyKDppYUN84liecjL/+GHvH1GkZrMF3nnGSWZ8cSeYhHSmDToRs2KMeEVSvctpMIwD4zAyuVHU522LmyZOiS7Ri66qbfdMYy/ERq3o+5oQlhRVKaLdZPNijbwYQcvYaUOkmeU09Yzl6pKLq2sWsxlucj8oiHISqrRZkO4OtDXnF+Js4UIkeHPeQBijhSHvnfCyonQKIYhflkqZgOLSECQuPEaKcmIWIS+WXF9d8ezZM+azOXmeUxQFRSHd09iPvLt/x2675XjY07VHyV8jUNc5VSYn5OLmihfPbxi6nof7B16/fsNstuDi8pIQI13XnS8yWSHLz+n7nhhFQX8Se5+E0qckWa01k3ccDnuqqsL5wNB1DG1HnmdcX13SVN9z6Ab05MRCB8UQJOaJH7T0MtanYA4VEv1ELl7pKpXQKtLYq3UBWYbzMMXA4CLdNBAUzIwYqE3RsD602NqgwkiIA4ocoxDm92JBXtRkRWAaRtq+Yz6bkRe3VHXBdruh6zvK+opxCGy2j4wjbLeRwXX0Y8fti2c8e/6Mu7t7iqriL/7iL/jmm28Z+xGlFHVVYVCsNxvGoePq6pr33/uAY9ty9+aO3XZ3JiMXRct8NuPy6or20DJaw7AfeP78mcR2lTkvXtzS9x1d14qf+3ePgpPpjMwWHNuOLK/ox0g/9hJwosQBVgJCjci7tCb4ScS/w8DkRqyGPHtqFPpeDsu+83TdxLHtWa0c88WMzOaJmA0QyXNLXcs9lzvH6CaJA0v8w9PmPBJxPpBNCjt5png6mhDcNAoORiJ9nrongYD/F8CktMmYnGwChnFgSIZv/uxO+fQhNqYpmmg4dUo90zTI6e5GvPc0zYzFbEV0kTB2hDzDDR39MEDwdO2B3W7DfD6nmc3Ic0tmxdNHK7A24+rZDe99+AHL5QV5UeJ94HjY83D3VrhTiZJwwrhU0u4pdfLCSm6j2iQWrWwt0YlqEDzeaHwmqTLOTZLKYTN6LcLS2EuEtU0x34urJbe3t9zc3HB9eZE2KyqNgBlKSU7cNvmenzg1mdWJJCsAfKYti2bBi+e3ZDZy3G94eLhju17z+LDheDziXWCz2fDsxQuyzPLu3VtmdU3TSA7b/tASMckCpErMewH0m6YmRtFF9qNsDzXiee6crM2P2x3RTSwXcy5WKx62B9n+JD94ncZ5owMWK52DIh29AUVIeXNi0xGDqOQl9Dp5kakBYxWVGiW8U1v6Ucz8h2nADY5tW1BFhx1aVnpkOS+JZY42OauLWxndQyDLcoqiPG9gbJYzWy6xmeV43HNsW6wNPH92zeE4EJRitmzQnWa3O3B9I+J4SbfO+cUvfsmb12/xPjBr5pR5TlWKFjSGwOeff06MkWfPX/DixQvWj2vhK4XIt998w6yZiVJgGpnPl6KxGx1FIcyhYRzwfkpWyJ5h7HBTi80qoUNoMCYyaypIqoPgAxoJPciMxhgI2kgoCeLVJqJ62WRn1p75cDIaa0KU+KxhHImIKDh6CNGTFxkXF3OqukibXkm5OUnVBLqR7eiokEIYIsEn0rec4lKIonTTpxjok/ljjDxRf36sInUqPkabc0KxOFOOScP2tH4EkjG8FLOh72Wjl2gMMQba9shu98jV5cBisSTPVxRHQ1UUAlQfjux2ex7ePXJn3zFbzHj27Ibnz55x++KGn3z2GU1Ts9seePPmDb/7/HfYvEjj3YJTQkiWWYzOngBzo89ETWlNk4BX6zMeE4jJ6ka80r2X09z7gHUW6x0my8jyjHKqWF1eMp/PWS3nZ++k9XrD4+MDj/f3vHjxnIvVJbvdjt/85jes12vGcaSqKkn1KHKKPAfEr2p5ccHlxYqL+Rz8yPrxnt3mgXFsIQqn5/LykovlCu9lhLt7946/+o9/hXeOly9f8vOf/Zwsz8iLApuLgSBKXhNr7ZNfVhATP6XdueOKIaTFQJCNXsiSu0TA6JMeIpCpQK40hVbiLmAU+OQ9H35geIbCY/DKCtdYRxweBycDICJQaIU1ljJo2pAxEWjHkcE5bPRYJqqLGTfPblnMKxZXF9R1TVmLc4BykyTC5BZlJEfPjQ7nIa/mmKzAWOFp3d4uafqB3f6A1jHx2jL2ux0Xqwu2W/m++XwuDql1Q98e2e92GKtp2xaVWZ7dXrFer+nbA/piSdvuCc5TViXew/6wZz5vuLu75/7hkbIQW+rr6yuUFiLkZrOBGKjrgue3l0kWE4hRMzrNbC5Y0TAFghM+kkwDQi+RBVaAmKcQWun+8zwnM/Z88NosoyrLxLOy6GSfQ6LeOB/OcE1VlRRFzniyyU5Ls2FwQrMZPEFxduVNO0JOIR1nAZlWGBTaPDm//S/mcQ4IsJtW6xoFPuDGAeenZHbnxN0y9XUh2bNMY0fX7oXdmwk1wCjHvt1wPKx5/vwli9kF82ZOU81ZNBO7Ykd0ivWDgJcvn7/gn//z/4TnL56zfnjgb3/9t+KC2HeUZUXTzJgZDdGDnyRFFYVBtoWnkUalwMKT/cr59zxpkzJlpJNKJ4fRYvZldMRrT53NaJoGk+K9h7QJfPW96Knquub29jkffbRgu37kb//27/nqyy/ph4GyFEyqqipxL3WOJs64WF3y7MUz5rMZwU+EybHd7dg+vON4WJMZyKzGO4mfatsON43MmjneT+TW8uH772NtxsXlBW/fvuXt27ds9zuWqxUffvQhy9WKGEu8c+RWwN/BTQQVqPOcXMM0aTb7gbFrIQaKTPhAxirmTS2x7kQyDXluuCwzZpm4O8YIQXsmJwaHmdYYZUAZPIZBW0YUXfAcx4mBgD/xaCOUWSER7kozxoH9YcdxHIkKCqW4WM25uLpGFXPK+ZJqviIvM0xRykiuDMPQEiZPHMPZCSLTBQqo6wVaF7ig2TzeM0wdZVXgQsQgqc9GiZXuB++/h4+w3R94fHhk7Dt02rZFPE2VJzH5xEcfvUcMkfa4pcw0u3ZPFyaapmEaRu6SnvK43+N9IMss280Dq9WS1WrJs+sbNptHhranWi2Es5VlxKhoj0fBhXw4/3ryaxPjw/ysrjuv9X9w758LGpytisSJ4rT9Rras00gMMWUEChFZKdlUh1NO3jRhlEq0I2SMS5hsOHl2IxfC6VuUrK/lsAqglD5rZ3/0IjVNAyGFGWgtJMuyshzaIZlXRSY3SUpvwpqEm+FAOSIO7/uUTFKAjlRVxrt393z9dceLZx/y7OYldTWDxrBYXPD++x/yv/sv/gvywvLlV1/wb/7r/5rddsNyOWcxn3NxsaJpTmLjkWnqcZPFTZYsEwcClAUVk/YtprStJ8HkuTidtlpK1qqo0wxuz+6eWZZT1TXeCTt7v9+x2+3x3lMUOZ988glZZnl8XPP733/Bq+9fcdyLtOQ0Gu/3e8ZRFOzNfM4nn33CixfvobXGe1Gt51nGYbvn8f4tYeyJwRGMl3baPaW51FXFbrfDu4kYJYtvmkY2X2wAqKoKk1vqqqbve8JaAgqsMbThSIwx8XQsrq7OEfaZ0dRlzjiNWKUwUTrUm5trvnn1GmsUlbUsm4KL3FJrhfaRKUSmFJ9dGCPLE2UYo+agNGsf2PQTu2GS5BCjkjZbuq1KO5p+QFvFfuxQKGpjWDQNL68veHFzycfvveDFzYUIYjONyRTRWjwRFzqiFqRWKcFJs7ykLGvGYSLPcopqhikK2uHAeOhEud8NtN1AP0jRr6qCWVNRNnOub5/x/Nkt64cH3r19zX63FbHxODFfzFg0JV3XYrShqXKuL99jGm95e3cvUIKPLBYL6qaiTdeBtZa+bzns95R5ztXlJX6Y2E4bHu93PEYJ/6zLWopuDGQ2xwTwzqGA3Eh3Yq1gckWRC06bEO4TSH0S7WtEtWCtIcvEbshYI77jY8CFxPWbRnKbpU5KdLpORaYphXpOElQi1CEvn+PENAWmqPFKAi9OST7xtPH7gRfaySbpSbD/IxUpH7xYnTiHG3tCdJK+opKVAxGVzPFD8CgFZVlA8CKkTHP08XjkeGipqwarapazGx4f1nz7zXeUxYKrT57z8UcfU1U1282az3/3Of/hP/wHvvzyd3TdnrK0WBOZNQUg8c1RBQgO53qmyeJcjpvEDiIkUD8qJZyrlAGI1FWe5C/qySlQnUzkzflTKwl+fHx8YByF2FrVFZ99/AnKKrbbLff3D2fb2LEfWczn5FlOWdYc2wOb9SOLxYKf/+LnfPTxRyitUqzTOy6WK5zzrNdruuMRN05YBXhH9APBRvI6F0LhJOTVzFrmM5EO7fdHLsqScRyp6xrnHQpDTs7usGe724p3epbTei8kUWMxwDSKcVtMdhtZlmGVwgPjMEAILOYzhsuRly9e8Ga95dgPlDojUxqjNF4F2jhyHEcKoDx122h6rbmfJl61I7vRM0UD2nJKgFap443TRBhHqsyyyjJms4bMwvsfvODDDz7g+uqKqiypmobVakaea6If0TgO+0ecd8LrGj1DNyTsRG5Uk+UM3mG0YXlxzQsf2Kwf8JOjacJ56/x49wqtxGSubGY0zYyL5YKmLHCDyJWO+z37bsvm8YH5vOH25hrnHIftmnHo8B522w1uCtjcUDcZl1fXvHh5xfphwzQ5bq4vebh/4O2bN3SHlqIo8Q72e/m9cx1GibtpN3S0XScC3hCwuagcTFr6DEOH1og7iBGPctnYPcm+jFJnTEprob/oRA/QGqGjpEXXNE2EToBu4ckFcb44Z1OKC4bSYuMcUFKcErh+IoTKrfRk3n2+x5T8TGN/5E4qyyyHw572sGcce5yf6LoDWkE3DgxDn4zrAi4B49Jx6eTPlJFlGXXVsN1u6doRazKi17x4/pJPPv2MX/z8lzgX+Prr39P3vcRMtS0vnt9iLby7e80w7GVD2LUspgGfW1mdgjC1pwk3DTh7YryKbMfmeaLqk7RFTyeOvIDitmmtSXlzScqSMvlOGwljDFeXF5RVxdD37HYb2r6jH8WBQWxQoDc9w2BRyLr/2fNnvHz5LynLgtdv3vDbz3/Hi5eyTVo/rPnNb/6esZdxsCxKMm3IFORW0zQNFxdzxuFIdxCu0mKxoG9b9ocDfd8KEJmM/6qyFIb8JIfFfD4X9rq2rO8f2e63dGPParnkYrkUKUQU/Cm3GZQlfddyPB4Yx1E4a9qS5Yar6wuuLlfsvvmOrlOUsSCzApxunePoPI2xzLURrFQpRjyPXct+ikwYMNI5yWpeJS+FSGE0ZQbzwrIqCvLC8OlnH1I2Jc+e3XJ5dS3b2nGijsKZKnKL8gM2zxlGI57ik5cObnLsj2u6YWKxuiDLMnwSxc6XVxiTM/UdXXuQ2HNr2e+3+GnkcNjRTYFnzw0hEz+vFy+e472j7zqqqoIQGfuJzaN4uV9dXrFvD7TdkaZZ0LWe7faR/eHI0E9UTS0j/iDdyLye8dlPfs40jOLxlVUEJdSQzf7A7tBRVRLyYbKcvu3oh55Ga+aLOTa36YAV219rM/KyIMsLJOCEp/QRTuNgPAFIT7hskrD5mFOUZbIGThzBk71SfHLNVUkSokFCT3WGMoEswpTwNMm4/AGUosW4URZZgkeZHxs4L4sMa8TAXm6KwND1ooEaJ4a+w3snc6+1Z4DVag1B/GM0ijwT/dnx0PHhBx/yk598RmYNv/nNb/k//x//D+R5zu3tLYvFAhUVhQXqHPvskrowrB8f6IYWNzju3z3gp4nlckFZlMQYCG5iGgfhuzxVF6EwZFka5+TFkXEgoyrr5Lt+MuSbkhGemEhbY5jP5xRlKZuwvme335yTYADKLCc3GaNNp5w2LBYLLlYXLBYLttstn//ud/Rdy09/8TNempf8w2//nv/mv/l/MI2OWdOwWi5kAxkjKi9YXqz49MP3cFPPu3evcUPHxXIJPrDbHejb47n7UVpoEk1Ts37cUJUVi/mCPM8x1jBOk0gnCJSFFBZDJLqJeSN+SeM0EoNDRU+uFbZpGLNMqBjO8/zFMwbnub254tvXrzmOIxpNyBRT9Oxc4OgBo+l1Rq5InuuByQUISi64AEbFM5dK48k0NLlhXhiaQnGxynj/gw/48OOPyPKSejbj6uqKq5ubNDIZwItoPIOFWVFWOf2xoz20GKVpbMbjes3u2BHIKOtSvt8IW70oS3QMDL10KVVRYrWin9zJeZGu79CxoG0PGGt49uwZmTGMfcvd27dopVnOZmij8H4iOieJPkPP0DsUlmlUfPX1OzCRqi5Zzpf0/cjbu+9YLoQ82ndiyGjymqaaY6sZm/Wah92RfT8xaxqKLGdWltSVSM5sbp4I0zZLCyGT3GPFxiidxZyCcOMJMz7RAtLXTsqMmHAukeUMP8gbAJ7qHSrxnyY3MjrxqgpJq6eVkDvPGz5IWQLm7OFmEMrKj1ykcpaLOd5NhOjw04i1FjeOYq8SxTEwxigGd06C5Ys8T09Q3vTMWv7sT/+Uly/f4+7uLf/v/9f/k+9ffYefHNPkWS1XtIecupBYcZRHM1FkiqvVSlTVa0mhEZqDMNCD9YToxCNqkA0eSES7OBikG8JoisLQNDPyPMd70RXudlJkgTNTvSxFWKuUmLI9Pj4weZ8EzE+nwElCEoOXSKnVBbP5jK5tubt7x29/+1tWqwv+8i//GcfDnn//H/89f/t3v6ZtRXdnjWzPonf4aeL25hk/+9lPmc9qvv/2G477LVWVYfOCx8ctKsTkbV1QVYU8/xCo64rHh0fqsuL29jb5WY9Mg+jDdFVS5uKbPfQdEFNRVxAD49BL+EBSApzG3GkSjs1+u2HW1FxfXjFfLHhzv+boI0E5Jh/Y+YDTmsrmHAJYq8gVBCN4SB48Pog4QgItbSLlQl1oVlXGrDRcrUree/+aZj6jaiqWyyuKqhHQtzzRKDRdt2foD+QWojIok6NzqBcFxmSMo2O2siy0put6Dm2PsSMAF1osSuI0Yo0cKjEG8dPSop7IjLx++2kgs5rlckmbHVk/3DMMElff9X0apSuE0hIYh44sz7luGo7Hgb73bPew648MIeKjQStD3ixonWLz3VuGfmR0I3nayGoltBYXDX6KhGNPYQ1VlVOUgJZgV5QsdvJcnFxPBUqJpiwdAlINgk7bQRJ2FYN0ltFzEiCfpgr/g/i5E+fR+emcFHOi68YohNKoNTo8/bs/BPCVSt2yVsnZVtKNzI8NnOd5zmw2J0YZ59o0Cng3QdBUVYX3hrYVQqGbHIderIS11jx//pKf/vRPKPKCN2/e8F//m3/Dev0g87BRWJ2J53K7Y7dV1EVGlkqt5MaJdODi8oLR9Ux+JEZJlh2GAauTz02q3jHNzEprJjQmeeNcXl2B0kzjwHazYXLyBskLn0amKm0wNdzf39P3g8zxUcifk5vOJw+IzKAsSvK8FNLqNPLq++/FVO3ykvdevuTYdvzDb37Dl19+Qdu1VEVFCO5M5mvbI3X1kj/9s1/x6Sef8c2XX/I3f/1KgivntQD00yD6OGAxm4GRCyDPZdw9bPeUeQFR8e2335DbjLIsmc1mwpyPkWkQL/X5fIZRMI0ju/0uOZn27DdbFsuFED/zHG0VfhJNmrE5OFjmBTfzFW/fPTI4h4uRKUDrRD40Rjg4h4qawmuGqDDKiGwmTBJoAWTGkxnFrMhoCs1ylvHei2d8+OEzVpcLbFGT5zNMljGbLySUoSzQRpwvm1lDi2O/W4tHu/P4KdB3QnspioKoFPNZjbIFjw9rPIGmLsi7EXzA6oyLiyvquuLh4Z7BSSy4HyZ0UYnLZ6LeDH1PZjXXN1eo6BiSU+nm8YH7+wNFKeEDy9UC5xx1U2GMpq40N7fP+P2333AcOnaHA0ZrqmohlJlqju46+s2j+Kb5QFkKOVmfcFGl8G5EKcMwOuJ2TwiOqiqwZcUpxv5pAZT+/AP9yQkXEoiDJMmK5w5JsgETHcHJJHFi00/jiEv+cacxLhHKZQQ8yeVSUTp78ycvNmPN2cVEa3UW+/+oRUprk6xtSy6WK6L3qZUMbDaPTNOYYmtkdZ/nOT54mqbhZz/7KUVR8u7uHffvHuj7nqoscLNGNkF5hnMejWLoOw6HHQ8JGyqKInGXFMoEisJS1xVtH1CI4wAhMgwdKLEU7seRIh8T8Jfz7PkVt8+f0XU9d2+FmCdv6BMdQRtN08yoqpK+f8JjvD/RKRKHagzJvE9R5hWz2Yy6ntG1HZv1VtjWGm6vrynKgvXjhs9/+1sObcs4jVxcXCZv8UiInmkaMZXmn/7Zn/Kf/LP/lDdvXvFf/Vf/J4rMcrlaolTB+vGRmJTmVgW0tUlzKBdBjCJuLvM8PX8p2kWRUZUF49iLWmDoEaBUy+rfiOVtdzxQlSWGyFHDcb9jv92xWC4Z+oHHxweapuHZixeM+wPsWrLBkUUYVGD0mikme1oHo3a0UeMc6KiJ0RAmRYYh04EigzyL5DmUheL5quJiLoEZL997zvX1M4qmpmpmaFNgbM44jmIJpEQknWXSyVZ1Q1AwecNXX37LV19+y7dff41zI5988iG/+MXPaTvxSN/sB7bHlvXjOz5+ccOzyxUZkcWs5Hjc4YKQLJdlibEWYw2FEfDYjQO9GxiHXopVlqOVIi8sRWFp25bNdsvu8QGICTPs0dEKayjATz75RCK1Dnu2uz27/ZZ9O1AUJdYYZrMG7UpxkzAKHcXNMjdi2Bisxgcn2E7yQzv59IvjhxYeYOIrRX8KBBGqwDSJxvQ01ejTRJBshZVWYsfkI2ESt5NxdMKNcqLfC1GnsU4Tok/RwIkwpdSZA6WRe8pok359EvyLGwg/fpEihtSm6bOTozEam4kf9nq9psiylLRieO+9W65vrjkcDnz++e/YbDY0dUNVVJRFIZ1JkHjzuqro+56H+3tiDHTdkbd3b2lb8dmez+fCB8kkqddaERM57xj6HucyqkqsWV3a6lxf3/D+B+9zfXXDsev5u7/7O2IIVFUtF3myT8mygiYJjnf7HW9fvyGmC+GEMcYoRnVidWKoZw1lVRMDHA4HseV1gdlsxvX1NdZa9vs9d998wzBMGGOoqpSLp+UlT87QvPfyOX/+5/+Urj3wr//1v+b+4R0fvv8Bi/m1pDS3R/HBNloSZzK5KCUhF7wXJnumjbTxCf+zpaXIcok2StpJrZADwXv6vkNFcSDIrBFrkfZIbiRJZxhGdput5Mt1PcYoNo+PmEkQvtxacmOZYnomEVn7I9jHOE14wAXw3qADVFlJlRmqwnGxqpnNC+pKLEVurq7OsqbZbMbi8oqinrHfd0zjRJ7LSa2NYBuSIDSiNWRFyeIi47ZzfPX196y3O2L0vL17YD5/zXsvX/D55//AX/3d3/P67pEQPJt3j/z804+5Xi45HDr6fs84HlgtG+bzRn5WhG4XiKkjr8oaYzOyXOLEh2nEhUg1m1HPG6pZjc2s2N1ESfN13tO2PeO4Z3m5ZDmbUZcVi2bBsR84tj1DP+GGiaauycsCTSRP9jUnGAAN1misyTEp7SiCHFbjQIgCWGujk6b0dOOmrL+EQUn4VPpKEDxTI53P6B390KdpYGIcJob0GU7oYYj4k/DeizpCJemLWHHLBHLejqfXTv8PPlWCSH7UIhWDR8WT+6UEXDo/iUBRa0nNNZpf/OJnFEXB96++49d//dccD0f5fmPIMyuAqVZolVGX4qg5n81QCH9qGHqiDzgf6LsuKf+hrmu6tqXIS6zWeDex2+0AOBylY9MKmvmCn//iF/zsZz/n/t09f/3Xf4ULwsquywodowiZm4bL1SWL5ZLdbsd3336Fm9yZWesTKO7SczbGMJvPmC8WHI4H3t295XA4kmcFy/mc22fPKYqch8dHjocjPtnGKg1ZblEmeVehKcuCsrzhP/3Lv8Bmin/7b/8tf/M3f0UgcnWVRo/HB2JwNGUlivWySiOoISqN8yLkzWxODOLxRPCJKKup60ri56NPF428h4dDLyx8YwQ/AIa+YxpHVHre00nqFAO5NaiyIEyew27LdJg47sUYrTCazom3VEB82FVamETviMFhUaxmS3KTMa9qVJiY1SUvXl5zeTWjqQuKomTWzCjLhno2RxvJmMuN5fmzW7a7Hfv9HojS4RiNyQx5UVAWmQS40kosVS3x9MGHZOmbERLW1NQNy7nFTT0vnn1IUS45DgHvejbbDd51YHK66XD2E8tzwTFNnqGMF8wnU+TaYoqK4J2QhfGYokTbgqEf6LqWvuvpDy3T6AGJlldWNnGLpmYxm+FDZJocx2PPcX+kLqVDCyFIwXNOMMRxoCgy6roSOyEiwzQxhQmtRrTqz6PhedyKInM5uRCcFAAmcZdObh2phjFOE0PX03anxKeJYXCMXkjNHoVzAeee0ou10um+VlilUVajjMJmopM9GUqexlB9Ypr/8TXqHxdpdfpHlRI/o74XMWNV1fzqT/8MrSPrzSN/8+tfs1k/MA6dnA6FsFa1iud1p1WaLLeQWlBrDXVdcthvzyOjl2A9ju1RDN26QaKhM+EsDeOID57jcc8wDPyrf/Uv+dWf/hlv377l3/xf/g1KaZrZjKKspPJPE9NU8slnz7m+vma/3/Pmt2/OUeb6xEL/gRKyyHNhs8/ntMcjX3/1Je3QE0KgqWs++vADMmvZrB/Y7fbnV/4kXH5i1YqWrSxz3nvvOXVd8vlv/4Hff/E5x8OBi4sLdBISf/fdd3jnmM1rARsT7mTKgqCgHweyFDk0Ooc1ER3llIwhMFum0M5xPK+QTx9NI5vMvu85Ho70xwPeOQHMnUQkFbkEpfa9bG8Xqwuc9xzbNmnFFERPmWfYEPCTT+O4Pkcf2QjzMuN21nC5XFA2NVmey3irA1fLmnlVcH11idKWrKglSDVG6rpm9I7t5pF6tmS5nItldZLrDENPrvIUM2+oyooirynLmiwvuLhY8fvPP8ena6TrOoZpxPvAbrsnzyzb/R5rZdnh/Ejw8OzZB1y9eAZKZF0xy4l5hs4tUUHnQftIVJaoMomcUl6kU25g8I5oa3RmyAI4r7l5tuS47zkeB0yuxKZYT+RZQZ5bok/vXVUQ3Ah4lM4oC7lmTWbR1tC2R9penofVCm0UVVlQVjluGunbDue8eMMnHpU+AdX6B1xAJQA2IFY4NktLKJ+kMRJJNU4TwzgxTIHJC+bo4ym1XHhQIjYW/3KbZWJ1rRSY+IOUpSf86qyh4VRH/vDP/7OLVPiBlYlKbOK8yFldLMnMc37/xec8PLxjmkaGrqM9HIgEju2RGD06QnsEl5dCy0/UBKMUVmsc4j2T2+xcI0KM+GT+FSJpC7enGweyzBJjoCxL/vP//H/Lz3/+U775+mv+y//yfy9Sk5PZ12CxJmMMkRcffchHH33E4+Oav/v13xCJFCnfDsCdeD3TSGZzLi8vWcwX7PY7vvnqKxGDBo9R8PLFMxbzGfdvv2O73QGKPBNQF61RPqYW+VTZ4fr6msVszn6344vf/479bsdyscIoS1XVDGPP119/xWa7wWaW0Y2M48jzZxas4ti36GSfYnKb9F1Jr6ZFuJlZKwXleBT84bwJFhtnW+QcDwemYQQiWWblfbAZxEGwpcnx6u4OYwxFUbIdtuRFRndoGbsJrQyrWUkX5mxHoSxEFciUpgBmyvLscs7zixnvP1uKH3peSGDrbMbQD/joyLKcEBV5ljOODm0yAop+HNI45Tgcj1T7mmY+pz8MTNPA8RDFs7sq6fqSY95SlRVKKZH/LBZ88MH7rB8fqEqxHynrim4c8Ui3d317yWJW8/btW4Zh4E9+9hN+8cufc3tzw6KZUZTiA35aoOgsI6IZh5FuEF7gNE64UXh50+mzHznsdmw3W+JmS9+NjHrPoCLqJCPbt9gsS9NFgfeePMtZXl4SPOz2e9a7R4yWblHkLDVuHBmGkS4K/63vPXnrUEh2pXeB0LaiPZWmPbnS/uGoZU4ylWQMGZM8y580uFEKkQuRfnQMo6THxNPFpBU6KTlI7gYxgeg6qTe0OS2WAqfJ7skh4R/38Y/L3UsdjrWWZtawWM7ZbNa8e7jHeZ8ShC02wn63ARUoigwVA23bMk3/H9r+M9qy68rvxX5r7XDyjZVzFQo5EQQIkGDqZmpmNNkkW93NoWepZUmW7Tc8/D7YH2w/fbHHkD/ZfpZbrddqqXNUdxNNMIGZRCByJFBAVaFy3br53pN2Wmv5w1xrn3OLlBoag+9gHFS655x99l57rjn/8z///4IiLdB6BhUnxJHGapnxSxKNwtHpthkMB7LLj8eUVYWONT0twbG0lq3tLZI04YMf+AC333EH5869xZ/+2Z+xubHB7Owcvd4sAHmW00ga7D++l8NHjrC9vc1zzz1Hnhd0u13SRuJn9IJwsYCH3U6X+fkFiqLgzNnTMrOkZfi32W7SbDbZWFvh/Jk3qaqKKElJk1TqfSs8HGcFmLfWEcUJ+/btwTlYXl4iG4/RWtFsNeW8dh3jsWZ7e0swpaRBZUqyPGfXrkW6vR7b/W0izzNzKkWXThjNSv7O+nI6TptkuVe6DMPAPoBFkXRfi7IU1+c4FtFAp7zqpwyL97e30WgaSYqthL1djBWmFDF+ixWXEmexPguLlKPpLIvtLntmeuyd67F7rsXCQgdUhdaamV63VoLQkRgGVEaROEXSSChNhbYxtcxxaUmTlDLPWM1lLVy+dEWGkRsNDh894u2cZsQkoDLMzs7S6bQ5fvwou/csYqqyBoRvuOkko9GY7e1t0rRBkRccPnZEuq5nz2CKMb1uD1sW2Kpibn6Wffv2MDc3S6PVImm2aDSbJM0G870Gke7g8BLVwd1aeylKlWIKCWqbW1uM84ztzQ22+hsM+33G4yGDQZ+NjXUhzDoxOcmznFZvNyoas7GxxapvxiSxuAnppCXu3liK3FINR6JDjpAjoziS4Wpnsc5gPFC+Q09cWnA4Pen2VYHzF3hNPpJURgbOI0L3MCJCQxRGhSWBqbziSKwVEcEizhMglEABcgQ1efEXH6SUs2gcjbRB5EW3slxmxawxwsnJC1CG+flZtD7G0rWr3h039mBnRZlZXLspAdlLRBgjtubtVoqpKpEZKSuGwzHD8UgASqDd7tBoNbn/3e/hnne+g3PnzvMHf/CHDAYDuu0WzWbba1tJy3jf3r3c+8570Vrz3LPPCJUgjuh0Ot49OaplWrUSwbfduxbRSnHl8iVGI5nJipOEZiNlZmaWfn+LqxfOM/IDuAC2qlBemzyKE3Sc+O4QzM7MMjc/x9r6Nlvb2zIe5PGATqstCo2estFqtVhc3MX21jZZPmbX7kVmej3OvXUO5aDTatLptqjKlMJrRzeihFYzxSaJdE2CVEYNlgpgai3oRJPlY1Eq8IYTGNmJ0zQVDfkkIY3F7ssZyTqUEX0wbR29douytGSjPrYocJUhso5GopiNFPu6TQ4sztGMFLGDbJSRthOSOKXVkbJNMKWEJGnKktWKTqeDsZbhaEQ16NObmWHXwgL9wYD+YJvBcMQbp95ge3ubo0ePkSjF2TdOMTs7z8zMLM1mk3GeMR4u0Gj4rmuryXhs5KaL5fzMdXvsWVwkiiPxhDSWm246STYaEuEwRly3h4NtXFUwGGywuXkNnCVJIq+ioXDGiFqBtbWOflDXUF7yOE7aNFst0kaTVrfHTDdi1+IBkiStPSCdcyQNYYhXWcna6jplLkqraxsbrG9scHVpiaVrq6yurbKxuUnupX1iHaGcxiGbh8MSW2+Xhnw/azzG6h2ejJe8rnAYv36VVxmVzq9k/5HytnPEMiQexpes8vOReGlg4SA638hCOcEUfTDaWfJNNKX+FwlS1rvsWu/E66wTBqlX6VPiV4Nywr2JdISpKvpbfdG5roQoOdvtMe4PKGLpULQ6HSyiuNlqNsFpqkpIZGmjSeEM2+Mh69ub3HnPPTzwwANcvbrE337lK5w9fUZsetKUvCj8AGNOt9vlvvvuY+/uPZx+803OXzjPzOysvLe1wEgmt60RWVUdsXf/XmZmely9fIWNtVW07/61ey0OHTpEVVouX7rEcNinLETPyhonwU55u3Xr0JUhiit6MzPs27cXYyrOvXWWyulaphUgUhrnFURdw/nv36q1sufn5+j22rz2+k9F2K3RrF1miiSlEclIiG5q4soQaU1ZGi/dLNdn54YlCylJEvJKRAiD8YPyutdpkgjnysF4OALrSNMmrWZTLMR9mZCNhmIrnhfEOJpa0U2b7J1ps3+uRzty2DJnaMY43ebytSvs3beXTrfnDQcSytLQ7qSeQiHdooaf0BccacRwOKxfk+cFe/bsIU0SxqMhC/NzUFpeePYZtI6YnZvj6LEjRNrSbLbIxn16M7OIZhXiclSVOB1L1qsUjabY2M/M9FDKY5II03o46lOMRzK0XowZDDbJxwOUMjIR4EewlDFerVK6nGUlmE5ZVR4PdBSl8fOegIp9O18MNlUU053p0e3OEiUNTBSxsLCXhcXd7Duyn1vfeQfNdgdUxFZ/yNraBpubW6yuLLO5scnm5iaD7W3W19ZYW1lhPBxSDDNvPCvrMva25saCcdLbs06cw4X9HbInVwcjYRUErpV3aApMKAfoAN96CkJoOHm5F6UUxFEdyHTtafm/aJCy3m/PhrLUs6wtZZbjjBHZ3QgRwyoLAaurknGWCdekrOi22xgjkXU4HAiXqiuzZWVVYZ2j0WzSyHLSluAIew8c4N533UdpHH/6538hKXKe0+31KPOghlnRTB0njh/nA+97H2+9dY4ffPd7tFstOp2OjPAoDVFwnBHNnwMHDjE3N8d2f5uXXnxJyKnO0YoiDh06wszMLBcuXPCaP4CT4FZVQS5YAPEwOtJsKvbv30+31+PihXPkWeEHMVsQe+5vwOVV0IMWoLTdEqB7ZmaW4XDAmdNnRM3SD3eLJXsFbUfckonyMF9lKkWpNVmeYapKWr9MCHdaCfagnKt3SfFNlJkspRVVVRJFCZvrGzRSuZE319ZJ4kR27jRh0B8yHo6xZUVUGTqI9NZ8mrKr26EdaVwxFt31vCJpxtjSMNweMNjq09/Yptfr0p2ZpcrzWvNKCMAFUZpI9zGOyIuCzHO7kiSm1+2Q5xmddoft7W1ajSZFnnP69Jvcettt7Nm9QCNJKPKcpNHg2rVlOp2ezJcpJ/BCJNl6FCeUVUUSMJ8kIU0bJHFMEqXEUcpQpZRVzkhFpBVUFWAKFMabDwgW56ylcobKGa92IeNZ2WhEkRdYi1eUTbE2IyUSv0pTko8rljeWueoseWXZGGWMCkOeW3SU0p6ZYWH3Xhb37OXwkRMs7NpNmjbYe2A/R04cF5PcQqg4RSaa6sPRkJXlZa4uXWVrfYNry9fob2/VhF6tNMG8NtYyfBwkubUKjHDQSrhYgi1NUiDn1y7+Z90U7imxwlCW1GW7SNrIcHHd3XubxqD/TUEq3JACB6t6FCSJZJq+2+6IyH4xFludMmem18VVJatlQSNN0A7WV9eY6c3QaMrU9fLyEnu0Im02KaqKza1ttja3ybKCPfv38auf+zxOwfe//wNeePElMTXUk5o5jhKqqmC22+Ohzz5Ep9XgK3/7t4zHmbh45LmoZyLYjFXQaXc5dOgwRz1O9dILL0k5E8tecfDgIQ4e2M+Vy1d4/bXX5MJYj5OAN5IoMFVIl+UCzc3Oc+TIYUbjERcunhdrpKRJhLCFTVXW2ZRzITuVwBFHmjRJWZxfZDgec2njIu12hzxTfi5Szn+ZRCLtmxcoa6iShGaaYr1Xm7PSgYm9NjkYaRPHEYU3U40jjfVtY1NIum6MjDEVRSEEWgdbGxv1SAPWUlQFOGilTcq4YiZukjQjhllOV0d0UOImYmVtNLsdqrKk1WiirGI8GFEZQz4W2/RmqyEOPUlMqy22YbGTsY40Ff7aVn+IMRk9j2d1ez2sdZRFwXg44vixY0SRYmF+jkhFOOPob/W5fPVNtrcHLF9bYXHXPMeOHKLX6dBst8X/MW0wv7AgpU9VYSqL9umB0pHvCINzHdKkSRp3wChGw3WcM1ROZPscYsDpjMVUhagFFBVllmGzEpP7deJk0BodoZVQRVKtxDoriUQl01bEiWIwrhhFhuGoZPPqMkuXlhiWJSqKaXc8ZxBxVN6zfx9zs/N0Oh327NrNvn37mGs1Wdi9m4NHj1Bl4gmQZWP6/T7Xri5x/q232FjfYHtrExoJKtYksSaOJGAp5TwGJWRnIXF67XrtRENdixSSGHrEk3GXSNf3pqj2OrkuznkfFz+kXLO1foFBCl/uXT+Tk8QJjTSlP9gmiRPGowFbWxu0Ww3SxUXKPK+jZ7fbYW1tg+WVZXrdDkkSs73dR0cJs/PzFH54tyxL7r77bvbu38/zzzzL+fPn6Q8HdBstyqLEICJbQsVX3PvO+/ilD7yfZ595ip++8rJMtCcp+TgjSRKx5Y4ioiTm8JEj3Hrb7YzGY55/4XnyceaFvhRz7VluuflGRqMRj//4MUbjkZRjiGdYSIODX1rlS9g4jjl8+AjHjh3lzNmzbPf7QhhtNKjKAmMqATWjCKuCbL50O6wTizClHN12i/E4o8gyFucXGAz6KD80bUwpNIxRhjOOPI5opynzMzMkbXHsQSnKqqx7ikEkQ3sbIudnqqIooqEUVhtIEq+yWtCIEzZX10mSBKyj1W6DFfKfclDmJZGCZpoQ9Xq00WRqTOYtwxo4nNDMRc2h1aSwFWkqJN+qFCvw7fEm21tbLO5eYH5+HjwBsdlssj3oo7Sw/xd27RIjVESTfm52jvWNDZQSCV3m5lhbWyVJIq8ZLxP8SSJmqGfOvEVZGNrthtykmxsYY5mbmyevDGm7xdFjx2qycDZuyXf3wIny3V7nxdpanQ7WZVQlOKy4epvSD9pKNmsqyMcVVeGwlcaWGlNab0UOKOtb99Gk4+bXV6ojdEOT6phWZGjpmFFUMSoMrSQR78JxHzvWqChisL3BhQvnRDY5Tmk2WiwuLHLo0CFO3HCcfXv3Mrcwz3AgOv+NNOXokSPcftttJEnM5voG165d5drSVdbWVhgPhuRVJVhx5ERn33sBKI9VBf9IOX6Pf/nAFL5PgBaccwLdOIGG8F6OyhOPf+FBSnkI1quUyQyfqYgjTbvZpL+16WfnHKvLK7TbKZ12i0aa0u10GI+GRErTTFI2RyLwnylHlo0ZjoY02210HHPDDSc4cuQop069wR/8p/9EnhXs27uX2XaX2I3YLsU6qsgL5ubn+ZWPfIxWu8lf/sVfsLW5QafZII4inHX0t/teG10Il7fdeQfNVotnnn6Gra0t0jT1xqMphw8f4sTxY5w9c4YLF86JZTdQZbmcUM+fUr5eK0pxJo6ThJtuPIFSiqefehzrIG02sbaiyCyKHJR0XuI4dH+8GSjaj3r43ceJJle72RAujHNEOCLlKMpcNgmvld5uNpjpdmh12lQObCYZowzLKkxlSKKJnrSxYRRIlkekRAcoBNo0TXHG+mOMyEdj2s0WkdbicDIao1NROTXKkkQJbd2lSlMhFSonNtqmQqmIXqcj8jj5CKUUvV4PsVMS49SxdwW21rJn/14xzcxE4rY/HDDo92k0G7Q6XckCixJsyqEDB3jr3Dl5PyBJ97K91fKKFzIUvrq2RqQ0rbTB7oUec7MzDAZ9Eo+NDgbbHDl+gpn5OcGrjCU3kt05ECdlz6C2VjZE5yztTpPezCHSSIs7r4MiGzMY9OkPttjqb7O9uUU5hmFpKHPIcodyciNjDcpaUJXMLipwxvopFYdWVjTElEPFmla3SS+1DPOcUV4wKA156UT3vYBG3CBpNcmNwVhHno24fHHIpQsXefLJJ4jjhN2Li9x48kZuPnkDuxd2A2Kwsb66xnA4ZHZmjuNHj6KUZjQeMuhvc/HiRa5du0Z/u48pS5/lS+YtNVvkCbRBLYr6dxLfbU0UtsaivbNxrLUMInv+1i9c9E4yKJ9J+ABVFTnWVCSJptNpkeUZWisW5hcYDDfZ3NiQ1nkkDq3GWLkphUPP7NwcWVUxynIOttrcettt4BRPP/scp994QyyVtMLmpbCLkwbjKMPiuOeeu3j/e9/PT1/9KT956iekiagJpkniZWLEg60oCvbt28c7730nZ8+f4/TpM7Lrt5qMsxGdTptbb72FdrvJ448/RjYe1fZNwGRA0zcFnCeyVsawd+9ubr/9dt548xTL15Z8QExBOaytCDrOCkWlFMaKs7JX/gIVoaKkpipIkNC0WqkPJi105NARvivjiBBD0zSOAMtmv492ll67QaNKyLKxuIQ4iyYWGVjPBtdezzoAn1opUi0qB8GDL0piMJa0kRB7oLzMK2xlsP64NSJHouKIJI1IbUpuDcSaEnGHKW2JKYxMCTQbqFYL65zYKM1GVFVJf9gXKyRjRNa3MgxGI5otkcS5fPkiBw4cJknT2sBDqQ6zs7M4HEmSUllDb2YG5xzLq2uUVcX8wgJHjhTs2rULrTXz8zM4V5GNhmxubrO+sYFV57m51eK2O+5E64iLFy8xGouHXa/XZa7XI00aDEcjtre2xDNvfYQxGThHGqW0Ww06nRbduRl279+HihOsVWxtSZm5tLTElaWrbG5sUBaZzLtWko02YleDztrfWmj8OI5sTiiIGwIZtFsJnbJBXpSMsopxXjGqLLkpSZXCRjLf6pQEWOsc2Iq15SXWl6/x9BM/Ik0SFucFkjh58iQnjh0VXmBRsLS0RFbkRHHM7Xe/g3ckCePxmJWVa5x/6xzbm+sCUSh5mokSTE2Scv5+CYojgRctInkWq5R3t7FEWmYHf7FByvMpHNK+NmWBtRXOCYjYaotNdBwrXFWQZwP64yF5Jiags34hjccZURQzGI3pLSzQm1vkyNGj3HzTrZw6dYrla9cYDgYerEWkVrAiNlbmqETxwfd/gJtO3sjXv/Y1lpaWmOl0iCI5vkba8BrKEiDuvusujt9wgu//4Adsbw9IGilaW7LxiD17dvPOe+/j/PlznDn9pojTa6jKUkw/PaM9gN3hHJRlyd333MPevXv4zne+TZ4XMp0/GsvOayTDipIYfDdFaUdUid4PShyWozglSZFOke8E+fhNEmlIYhRN6STWImQiUTPOcvIiQ2tHM00wOMbZCGcKVLtN4sdepFSttxmx5PJ/CsoOERFKOyyWKI3RVtFIE2xZUhYFOsJLwvgsLY6ExmANNlLEJFAWqDgiVZrSCsu9spKZpWmDOIqxWPJ8jDExaZrQarVl+iBJGQ7HtDsdqrLi6sZVZmZ7NBoNNtfX2LVrF400FYOO0Zh2p4vWGussrY7YdJmyYl+UsLa+RhxHzM/NsW/vXgCiRBF7PHHPeMxoJA46eZ5z6tTr7N27j5M3niQrMlaWV9ja2iLLMmbmZuh2u+zes0ikhSayvr7B2soqy0tXWFu7RlWNsVWB1oq00aLR6NDtzjI3v8ihQ0dotlvkZcF2f5Ot9VVWrl5hY3mZwfYW49G4BrMjpLyMk4ikoVGx9dwlK3gRMY00omrE9FLDuLSMipJBnlEah1MaEzmslfm6ICOsEaVMwU0dWxtrvLKxxisvvUgcx+zas5tDh45w+OhRTtx4IyqKuHDxAtdW18jznN2LCxw/cRytNJcvXuTypYtsrm/U84QhE8UhjRnlcSs8NcH6CgwwgNFSSmothrJvK/a4HSyv//Lj6iuvSxZlHYW34ZFlLYBYmDHKxiOuXbnE8spVsmzE8vIy62urNBtNFuYXuLq8xGg4oigrmq02H/3ox8iKnBeef4GiKIijmDzL5Ge81Eur3UFFEZ2ZHu964N0Mt7f57qOPesfkjCiJabfbJFFcq/41mk3e8+CDNBoNfvSjH8nIQJyQNBo0mylHjx3l1ttu4alnnmZ9dY0kiWg2GhhTURZihFAUYsHlAh/K6+g8+OB7iZOEH/7we17xQTz0mk35NfEjAnEST4KDJ1NKkFIoHZHEDZ9FyXyYpMvOW9JLSW2MEOVMZRn5Gywbj2UXjhROWVKt6DYTWmnEXLdDr92m1WgTeUPI2OMEypuh1gfkrFh2VRVKa6qioMqlu6mNgNPZOCPRmjSKqQpxdhZemFBNhFoic2Qq0litqKzBKpnfCpwZmdwXnpbDMtPryYhPlrG4a4/YficJZVUyGA4oqoKZmRlmZ2ZJGyLyNhyNuXjxIou7dnP82AniNCHPc7o9mf0cjqQDmmcZkVYUVUm31/WmAl7FwoqFVxyJvbhB1Znk7r376HY75EXOxvo6o/GQbDSUhgMiird7915m5xdoJA02tra4cmmJq1evsLmxQjbcIsZiy8LPQDrQEWmzwfz8PHv27mHX7r202zPEaQOrFOPRmO2tLTbWVllbXWF7a51hf6uefbRWslacNzwwBlM58spRlJasrBiXhR9mVlTWYRwY68eJnXDBLKEjJ9deeS5hXpYSxKKIRrPJ7j17OHT4CEePHaPdbpHlY9ZWVxkOBvS6PdqtFI3m2tXL9L/svRMAAIfWSURBVLe36G9tCWHW451aURufiChoQEYnyY7yo15aa/7tIz/8xQWpSy+85DtSFmPFHNI71IGFPJeOV5aNWF66wmC4TZnnXL16leVr1yiKgrm5OQbDIUtLS5w8eZJ77nkHZ8+e5dXXfkqn1ZZOnam8+8oQZ2UeLGo0OXrsOO+85x6ee/ppnn/2OZJIFAgrZ72bSUS71RY79UbK/e+6n6Isefa5Z4nTuHYkbrVbfPCDH0RrzbPPPkNe5NIVQzzksMIDGw1GYsVVFB7YFsLhL33wg6ytrvKjH/1QBMrS1DvDNoTnkyZelC0mTRLhAfn2rLg6y4VLEtEeKktT++yFYUznJlo7zjnKopJhz7JglOdUpagaWGcxtqSZxvRaDTqthJlWiyTSNOIGnU5bglTkwU8/qIPzOy0OjNA+FCLy7yq5EZy/IVwlz0hpqlKIjsqXpsIXU/5YZOLfeJKg8QRRtMZYcTlRkfb0AkVR5HR6kl3HSQJK0Wq3hV9kq9ozcWZuHq01g8GAsqwY9PtsbGxx8uRJFnftojIVK6urLCws0u3NYK2TaxDHDLMh49GI2dkZGn7ExfgulUNLi92D2JUVIxHlYYhuT3wL19dWJPtZX5OJiTwXqGJ2jsXFvezZc4SZ+V1yM6+vcPXCOVauXGK0vcl42KfIcops7JUFKrLC4KKEtD3D7PwCe/ftZf/+fezavZtur0uSxpR5xtbmOktXL7O2vMzqyjXKfEyVZcKvcxCkWMrSMC4KKgeV52RVVjY6i6hlWif0Him/VG1wq7Rv3CiZwwsbsTGOJE2Zm5/j+LFjHD16jLnZWYqiYNDfYmt7CxzMzc3QSGLWVla4evUKo2Gfqiz9OlMSGKcCVfidAPHSOPj33/zRLy5IvfWTZybDsko6UkaU13HWkmclWZ7R729xbekK29tbLF29Qj7OfPASEbKyKjl29BgHDx3khz/8Plub61RVSaw1hw8dxjnFaJyzud1nMBqh4pj7H3wvJ46f4Mff/wFr11awZQlKBOdLU1L5QWQdRXS7Pe699z42NzZ5+ZWXaTQbNFoNtNbMzc7yvve/n7fOnuWnP/0pUexNBzzzV7INqaOzcSZDuFnGeJyxuGsXH/3oR3n1lZd47dVXUFixck9TGg0ZWWh4GkCw/Uo8C1w0ob3FuydUJklCUYrrrEJ7prz23RQhyaJ8kCqlRe4cFMaIo6x1njpQkMSaTiMl0o4Yi7aW2e4s8/NzNBoxSRR7op6fiPdtEBBSYuC8OCMu0WKnLlmWMxZlpNMTunMY+ZmyLITOYB0qiukPBwIyI6l9EseglSg2aOXlXCBONFk2YtZra+WlmEeoKBYcJo68h6Om3emCUt6JeoytJNONdEy316XVanHh4kWWl5c5evwE+/btY3FhkaIqaHVaXLp8icpULMwvynB1jdFFXhQx9hLLMUUI/ojmVmFK5ucXaDWbXLt2jbfOnqG/vQ2mBCsS01VhUHHKrj17OXDoKPMLu2i12+TZiNVr11i6coXLFy+xsrTMeDhgPBySZ4JPWWtEeiaJidOIRksko/cf2M++/fs4dOgw7VaH0ShjeXmF06ffZG19g+H2AFuUqEpca/KipHLWKx573pa1FL7sM9YJkdNaMfEEud6omq40YYUL7hiCNyiMsTSaDY4cOcyNJ2/gwP79WOtY21hlMOyjFHTbHfJszNUrl1ldXfbD7RaM6L75sCEqoUpoOzrS/MfvPPmLC1Kvfvf7xF58KzBsKysyvs5Ysjxnc2uL1bVVLl04R1FknD1zhuVryzQbYk65f/9+7r//XVy+fIk333iDONHk2QgQt452q8NMdx5jFINxzrgsuf/Bd2NxfP+736WdNklUhLIOY0QRcns0pDQVDsXi/AL3vus+zp07z4svvECn2xF/vEhz8uRJ7n7HO3jmmWc4feY0kW97p4l4i4X4G3bXoqgYjkeMxmOarRaf/vSn+fGPf8Qbp14n9XrorSSh1W7RajZ8sEq9vruUNkmcyKClFsVEGfSMaLVbYj1f5H5Hl+FfHUl56JwcR2j5CrYkSbMD320SB5+qKrCmRNmSYjyilSYszs4yPztPu9UkTROSJPKT79YHqfBdlZQSKsjMSuByPpPCiaoCnkMl5Z5gjFVZeVcgVS/y0Xhce8IZ54OUB2Z1pNFxjFPWD8wmPnvUjLOcTq9LHMcyvmEN3W5XdlufMVvvpNPf7tPr9QSXG41oN1usrq5y/uJFUBH3vPMeDhw6yDjLMM7QbDc5e/YM83PzzM3NiVFBLHyyTrdD2mgRRbFYLHkcrzQSrMZZzmA0Jo5TDuw/QKORcvnyFc6deYPR+gpxZLG2pKoKytJQ5FAaaLQ7LOzZy8EjR9mzZy+93gzjccbl85d486evceH0KbZXltGRyEBHkQJtUdoQR3GNGzabbTrdWRZ372PvoQMs7NtLuzPH1nrOtUtXuHLxIitLV1ldWWY4GvrWmiVOhEhYWZE8qqzFWtnUKyvO03bSryf05iTbDqqZAdSWdSv+egIXtNtt9u7fx8kbb2DP/r1EScRoMJSMXEFR5GysrbOydI3N9XWKrPBmsWHSwnMLUfzxD57+xQWp57/+zdr9FqXQseAcRVlS+uc4z1hdXeXNU69zbfkqeZ6zsryMMYa777qLO26/naefforRaMTC3BxlldeDrWWZU5UWU4HWMbv27OPBD3yAC5cv8e1vfwtnLe3gohIJzb90lv54TFaWHDp0hDtvv4Pnnn+WCxfO02w2iZOERqPBiRMneN9738u3v/MdVldXGedjiiL3oxgN4cYgPI9moy38psqwur7Gnj17eeCB+/nWt77FysoySRxhjcGWBWksaoqdVlOCVNqog1IIVNpLumpNzWy2SNfRWCPAdSQ26MZB6SfOQ7DUnrIQ+dIxBAXnhAYhwW5MjCXCMT8zS6fZIE1Sr+ioaaQJzWZCFE2l3cqzf30g1C7gBU5K3rKaYAmeqV6VpZQveV6LAIaRFuvHIYB6F03SBtYYxuMxURwRpQlxKrZK3ZkezWbCaDRinJfebkkcTmSKX9FstUUsUEU4axn0B2xtb6G0pttqMxqOGA2HZFnmz01Ep9tlfmGBtNVgbXNdMKkkZm1lmXa7Q6crVIgiK2g2myTNFp1ul1a7LQYEwKjIfAmsGPQHbG5sMR6Lltktt95CM0156403uHDhLMPRFlpZL1Ej75tlY0rjMD7b73Q7HDl6jBtO3Mqe3fuJleKtc+d58fnnWL52ldFgC0xBEjniaJLVBEG6vCwpsag0ot1b5MDB2zh8+AgL8/OkacLG5joXL17g7JkzrK2usrm1SVmWHtOEyjiME8VN4zMrWwcowlWuN8WgVBuaT8GFWnnYQjp3MnnS6nSYX5zn8LEjHDx4mFazSZ5nFHlO1h8y6PdZXrrG+vo6eZ4L1cKD+Eop/uyHz/3igtQLX/+GCL8DlRW7qjiV0YLIt7GH4xHLK8u89MILXLx4Xkhyccwdd95BmiT8+Mc/ZmN9jT2795BnGVtbW+S5dPuCHIvWCQcOHuDjn/wkP/zhj3n+ueeEvp9EojiAot1oid5RVVE4y8lbbuHw4aM88dhjXLlyGaUV7XabOIq47777OHToID/4wQ8YDkfkec44l7ENrASINE1lN40iOu2upP55wd69e7npllv4xje+Qb+/RRxH/qYUtcRGHNFut4UP1mgI/SHSdXCZ6OiI8HzakLKzP+jXowVBuTDkwLU2tNfj0X7yXPssNkivWiN4lGAIJZFyNKKYTqtFEkVUVUGMotlMxV0kFlfaJA5lpSff+UxOh2ClhaxqTSUMbOXb4lCXe8YbndYDq9LzpqyEtlFVFdbgTTAz8twLsqVSzimlRF8qEfMD48I4hZWhWwXD4Ui0oRbmhJPkN5H+YFATCdvNNttbW16jKqIoKhqNhigSJDGVrRgMBtxw8gY67RavvPIKcZzQbrfpdbssLO5iqy9SviqKWFhYoN3u4hSsra3hvMqpqUpWV1fZXF+nLCsOHjrMzTfdglWa02++ycVzZ3FmTKKsTCPkQs0R9QFDVRZUlQESkrTDvsPHuen2u9h/8BDGWc6eOc35s29x+eI5ytFQHKJjuUbKSSZrTEFhSgpjKV0LoxSlMbTabQ4dPsKevXs4eOgwjUaT1dU1NjY2eeWVV1m6usJ2f0BZlsIUV5rKGplhVao28ZSHZGCTIDURytNB+8k3j7SWe4co8vpukgXu2buXG244yZ5du2nEMXmR0+8P2N7aYunaNVZWVqiqos7a/vLHz/7igtRzj3yVZrMholfOYj045xTM9HoYa9na3uDa8jJnz5xhaWmJ+bk57rj9dl599VVeffUVP/0f0Ww1UUrLgOTWlmewJjgUB44c5lc+/nG+953vcvbNN6FOSkUl05qKdtqi3e5ilOLmO26nMzPD177+dVZXVmg0RKa3223z4Q9/iE67zY9/9EPG4xEWLUEqE9E6rDCZ0zQlShKiKKblLbuPHjrKjSdv5Gtff4R+f5tms4n2kh9hMDeJRZG01fJBynfSJgFqMvMU+dKwKAqGw2FNZ7AeqDfGgJbupFYTrCx4leH9yggsZc8jUJ7FrIFYiVSGqcR1pt1q0G43vdaUo9VskHgH2zgW3EscPCJRY9BKXu+F8pQOi1gA98oYTFlhqwpTSalZVSVY0AiwHjK9sjTezSenMpUsej0BbdGKRjOR3TqK68BjrEFFkWgjWZFkmZmZoarEsSbPM/BBrDvbIxuJHVWj0aQoSsbjMQ2PMQkeY2i1Wpw4cZwoinjzzTfD0peh5KPHMNZy6s3TbA36aBVz7Ohx5ubnuXr1CsvLy+zatYtOt8Mbr5/i4oVLjLMBjQSOH7+Jm2++nf7mNhfOvc7q8mVGowFVJcPbOId2ghEqEAVVC0Ulc3rt3gyL+/Zz8uZbOXjkGMY6+pt9zp05zbXlJdbWVnBVRT4c0lCayEmrqrLiKpwVJXlVYrxLsXHQ6vTYu/8Ax46doNOdQUcp24Mhr596nUuXL9PfHlKYCslX3XU6dFL2BZ0p7y0qxMtQGHpnGT/LgIo0aZrUiUYIdo00Zd/ePRw+cpiFXbtwSjHKRqyvrXP58mWWV1aoypK/e/wXmEm9+LWvkqYp4/EYtIBpg+GQtNlkbmEBHWk2NtZYXVmh3+97wbSU5555hlOnXqfZbLG4uEASadbW1rzuc0FWFETeq+7goUN8+GMf5YknnuCN108RWa826SxOi6NuaSrSKKHb7XHXO+6h2evyjUe/zbXl5VpquNPp8PGPfZRuu82Pf/xj0jSmLEtGWS5Bapz7WTgrGjhpWkvzxlHMzTee5B133M3DX/kK49GQOIpoNlLSJKGylbR0XUUS6anhVCFYxknopnlJD48fRYmIBI7HYw8KKzmW0RgTFCb8nhbFsQ96qce1hLago8gLisUkUTTVWhfZm9CRwwo4rzHoSNFMheiqsJLRpSmNJCaJEx+0vFdgpD0nKwgc+j3WeTF/z9ESYF3OX1DHUE5hKyPW3c4xHo28FZLx50DGQpyCKBJBv9grTUQ1RhWL9ImOqEqDsVIydjsdKlPWoxdRHHNtZZnZ+TnP+pZyb25uXhyAioJ2q0WUJF5LytHtdjl27DjNTodzb51l0N+mLCWYHDt+gsXd+7m8tMybb55h2B9w8NAhjt1wA05rXn71NYx13HLjzSxdXeInT3yP/toFnIGFuT0cPnCYo4f38dTTj7OxvuZLVgCLClpMngMnZGZLURYUVUllFaVRpI0uc7t2c/LmOzh+4iTzuxcZjEacPn2G1aVrXDl/kf7KGlVQ4ECchPFa584hLjcotkcjSutQUcqevfu57Y47ufGmk8zNz3P+/AVef+MUly6eZ3V1xUtmq5p0rLRCexOsAFPI0LFXO5iScgxZlqwjr93v142ONGUhTZ7u7AxHjh7lyPFj7N23l7IsuXjxEm+dPcvv/NXDv7ggdep736pncbTSGFOxtr6OiyIWdu1GxxGrq8sM+30aScpoNODRRx9lc2tD3DWiSMBQpciyjH6/7wE9qY9PnDjOQ5/9LN/93vc4c+YMaZSgrLBvi6rCRZpxWeKimGarxTvfcQ9lWfHkU0+zsb1F34t/dTotPv2pT2GrkueefZZ2q4VD+DgyVV+QlxVFIUS5ibOq3PC33nIr99x1N19/+GFZYL5UazQadDttwYFKsZKPYi1W5V79UHk5iiiOxAw1iupp76SRgII8K8Bzc0bj0URsDE9c9WlwFEnXSTKqIAUrKg5x4lUJYtF+iqMIEHJlkGixlQFn6HVapI3ET7k7Kc2LQgwuuh2ajVSwGW/AIOJkbjKn6TcJuRO8gmNZSaaAF/ZXorJqKulWVWVF7h2dnceqQmBWkXSNhJ/jpURSoWzEsWBCkTdaKIwRQcFU3HWGQ9HL7/W69Pt92p0ulakYbG+RFzk3nDjJcDjyMtUTM0pjDbGOSdoddh87RqvZZOn8ObbWVijznCwvWdy9j5tvvYuisjzx5BOsrqzilOaGm27ipltvZ2l5mTdef4ObT95IEise++H3OXPqDVRpSJWi3bRYm9ccN6sEgHfWImb2kpWGm81Z4zc8Q2mc5zgpjNMUVtHodjlw+DCHDh/llptvZdfibi6eO8eLL7zAmdNvMuhvCa3GY4IO2VyysqIwlVifo/wUm3SWd+/dy11338073nE3nU6b1187xfPPPceZ02dFaw0fYLyBp2/6SmnnR7dq26o6w5ZRp9TTSHwbpQ5YlTWUxlJaQ9xocODAIW6+5WaOHj2Cw/Hr//L/8IsMUt+gLKsabzGVYW19E6NgdmEBpRUbG+vMzsywdPkK3/nOoyLFYkqZzxsMiKOIRqNVs7adEsJdp9vhM5/5DK+8/BLXlpaErVxUIrqGj9hxwnaWkbTa3PmOuzGV4bEfP8Yoy2s95jRJeeihz7C+ssrLL7/I3MyM3BBZxjjLBPi1DmMceen5QUp58TPDXXfdyW233sK3v/4NXFHS9hlHHEU0mw3hzpSlt1QvRRRMR35yfIKrhEWRJHFNLWi0Gjgco+GYPJeMLmQsshPa2vdMgHPJMNJEeFiRqOej68xHE0UxSRKLEKFWQgD010trTbfVJElj0frOM8psTJ5laAW7d+3yhq/ilFMPjarQQ/QPG0igApJbY+u5RlnU1KWr8QYOVVVSel6QtRVVJRLHWmtPMJQbOLS6VSwlZhynxJHQM5QWnCNYWCmlGY9HPhDKsQrmNWY0GpDnOfv270f7DNVaQ7vdot1qiR7/OIO4QdTtcuToMXrtFhfeOsvK0jWqomRclBCl3HrX3Rw6coSXn3+Jl599kdVrK8zOL/D+D/0y+48f4/mXXiJOYo4cPMhzTz7F808+RTkckEYZWvnvhBRTknGLAkLgC4VumrNSilonHDLRD8dTBRS5U5RO1mmcpOzbu5/bbrmVu+68C63hzJk3efHFF7l46aJ4WyoZ2q1qDpwf4VIT/X6loKgqkjjm0OEjvPOd7+TWW2/DGjj1+uu89OILoq9vJpK/nj0zgTB8jih0Atmk4tpkZeICI8RkJby5ANR7zFVpza7du7jt9tv4P/2b/9cvLkg9+/B/FuODqvKdGGkXE0WkrbYIxmtYXrrGW2fe4ty500QRjEdDVq4ts7m5QZKktDtdrLXkRYF1jma7xQc++AHeeussZ06frv25MJZxJppUjaTJ7MI8mTUcPXGSuNHg8Sd+wvbmNpUxDEZD0rTBl774a1w8d44XXniemV6PZpqSeKb0OM8oKlMbWZbGUAU6v4M777qT4zcc45tfewRlDN2kSaI1zVRUHprNlF6vi3OOvMjFpEC5GthWXmmg8hbtWsuwa5I0ahXGosjZ7g/IsnF909dP8ECrx7B8dieBKiFJozrgRVFE6lPs2JdtymNJSRzTbDVkRk5rjBGcZmNjnfFoSLvVotdpU1UFczM99uxepNWU7mQcx0Ra1fOKMoslJYvQHlydqUlH0O0IaLKb+3NQituI8fhMkJ0W/M3KrIIfGRIuVSXaWklCHIkaqglByh9bVVVSQnpNcB1pmQjw8rVRFNHtdCjLguFwQOXn+GZnemxsbJIVJY1Ol/bsPDNzCxw4cJCLFy5y6tWfMhr0yfOMCjh8/AbuufteVOF44keP8cKLLzGsSk7cegsf/OhHcVqzsbHBrtl5li9e5vvf/BYbqxfQThQoIqUJYrkyk1kjQLIxhUCFYJJVJRt2aB5Zp8gtlD7gOKcxxqGdptFssLg4z7333cvNt97CaDTiscce57XXXmcwHFIDlWHz87WZ3pHluLqr3G53OOwD1pFjx1lbW+PF51/gjTdOsbW1IcKQsZ9akFYv+EZKmDuMtGDNUeg+h1/dZHVIjPZNpVi4gI1Ggz959Du/uCD19N/9FeDB06pCe16PA9JWm/mFeTY211m6KiXftWuXWV1ZQuFYXl5mPBx5zMGRjccUxpA0Et7//g+wdG2J55571jujeNDOgTFy07abbdq9HgePHqXR7fL0M8+xvrntzQtzmq0mn/vc53jx+ed55aUX6PW6tBpNkjgmjRKsETv2YZ6TVRWlQQT5lRiQ3nzzzdx080088rWvYvKMVprSTRrEKJJYe8VICVLadzJH47Gku1rJkLASUF1s30u/ANri85em5GXBYDDwBgmTMZsQoPRUNgYT2QutPe6VRsJS98FLnD5iHwAjn3bHzM3N0m63KMuSfDwWLe3hCGsqUYdwlu2tTVrNlFtuvpFuq0GaxLQ7Hcn+Iu0VKicDo8oZP5MY9LN81ydEeKayKT+HiBXyZ1XmdfCVQGO8mqkAybLsnZSGOE/diEhTsbUyOC/D0vA3dFWfP+0VGoBaDbLZbAKimDr2lmi7d+2i0WiwurZGf5TRW9hFZ26RZmeG48dv4tRrr/HiM08wWF8hz3LywrK4Zz/vvPc+Tt56C2+cOcv3vvcDzp27SKwjHvzgB7n//e+nyHJiC4mK+MEPvsGzP3kckxfEKvJlkcehrNzaFj/460swp6znulXhNhb+G95s1dugO6e8cJ6/KtpRlBWLu3Zz/wMPcO+97yKOY554/HGee+5F1tZWqYxIDDslxNQI9TNrTD7PCQ3OGmZm57n9jjt49wMP0Gw1OX3mDV566UUuX7xEmReyzpTICukAsiOzoXGoKLRYWkmQ0rVagr9g9RpO4gZpI+X3vvrILy5IvfTNvyeKYoyxAv4aaRc3mk06vR79wRajcYYpLdubm6xvrLB87Qq9mR6rKyts97dJVcxwo8+VtWUybbnvve9BFYbvf+e7HmOJaheLmhXtHO12l30HDnHs5I089pOfsLa1TWVlWFkBDz30EK//9FXOnHmDhheBS/2QcaQ1trIUeUF/nJFVBUWAWICbbr6J22+/g4f/7isMR30aSUwrSWinCTFivRVFsoN1Om1/o5XkRS4n0JPfQIwaS6+bpLSi2WjS8N2/0A4fDkZUVYk11guHTUTCCJQFJtmUqJ3KziQZVYNYeR33SIT3o0j4WgsLC1Le5hnD4ZDCl5X4txwNB2ytb1CWBSeOH+GGG47hTEG31xH3XBXRSGRHDDuu4A8hOHhCXp36ux023mHuUHSrVA2yT2eI0xiK/JWUfmVVCom1BvAlQFfW0EhTWp2Oz1SruiOaJCKcqBTigD0l/+F8I0GIpBHzi4vMzs5x4eIV1jb6HDh6nO7CLvpDw1133sUzj/+AJ3/wHcrRCGscpYNSKQ4dPcqvfOKTNFodvvPt7/L0T55mdavPgWPH+MxnPsPJEzdQlSJW98qLz/P9Rx9l5eoSsVOkUZAi8VkU4IIhgRKukbXGu1FPzonFz+CFdeqkDLQOjHMiHAcUHn6ZnZ3llptv5b3vez+Liwu8/vopnvzJk6I1VZbS+Y0jwtKahCk/HhNpGaMxghGnjZTFhQXuve9e3vnOe9ja2uSlF1/klVdeYdDfQjIpoc+ETCqaClLTfD6Uv5ZRJA0y7/XXbIp217/96//8D8aetx2kXv/+t/xu6si9ZG+jKfrXm9tbDEd9kVR2ggFdW75Mlg/pdbtsbm5ybfkaLq/INgcs97fYffwQi4f28+T3f0y+PZBd1I+QRHgw20f+bm+OW++4i8tXljh1+gyVgtwaqtLwqw89JOnpqZ8yPz9DM0586ump+ChMKRnOKBfQuwSMggMHD/LAu9/N3z/89/S3tkE5kkjTShO6zQaJ0nVaGyexL98if6NUWHyN7akElbXe1VVS4iiKaTSadNptrKvoDwaMR2OKQgDTSMtIRBTrendVNTNcshORsxUdqkhpGo1W3UmRclORNhMOHTxIFEVsb22L1pUxvrtW+WuWsbUuna9mM+XEscMsLs5hq5zdexeZm51B61j8/OqRHC8T7awPBj6zuy5I1cC/D1LghI/jqRAhSInt987lJuBqWQcpkM+wxqC0SB4naULTm34GuoYxhmaz6Vn7VvTlp26OUEZZYymNCK3tPXCQ/Xv38/Krr7O8tsHxm25mfvcRytJy4sghvv7w3/LK88+Sj8cCCccJeSk43wc//Mt86KMf48033+JP//zPuXR1CYXivnfdz69+/vPMzc/KYPz2Nt/95rd4/ulnKEZjEh8cXB0dXK1O6byjS41L+jLMOicaYVaHV0qQQlGFbncd6VRNPNU64tDhg3z4Ix/lxA0nWFtb4dmnnuaFF14gy0ZT1AH/2smMgb8YTKAHJW5LvV6HW2+7lfvvv5/FhQVOv/kGTz/9FMtLYk4RRdILVFqMISI9sVgPaziKU1HCSFM/jC92YWma8m9+/z/9g7HnbQepVx79Gv3BgDyTG6DVarG4axcbGxtUpsK5iiwraCQtwHH+4llWV5aYnZ3FWMOVK1fIhmO2VjfYdWAfx26+iW98+1tkgyHUWuFCKtQ+IjcbTayD4ydOouOUV19/g81+H6s147zkl3/5l1lZXeHF555lptcmTSISXwphBciN/BxcXpaM8oJxWZAbw+79+/jwRz7CV/7u78QjzVi0gjTWtNKUXqvpvQFl8cdRRBRH9VyZw2Kd5/w4KS+KsqL030XkVDWNRkPY5KZiNB6SZ+KdJg45oZSL65tLgMedjrOxz6hiHYnEbhTjESB6vR6Lu3dhnWU4HMq4kLf5DgGzKEqysbCAXWXp9TocOriPIh/RbiUcPLiX+flZ4lgknj3/dNLhcWIcGkUC1gffNlnwk25gKF/rgOWhgcBId55A+PODVFUfs9aKsqzqWylOEro9j2XmvmvopwWyPCOKRakzzEYG+/DxSNQ2SiOE0aKsuP32Ozh4+Ajf+/4P2RqMuPGWOzh27EaUjtm1uMif/emf8PILL9Z5pAMqW1GakqPHb+ALX/wSu3bv4S//6q955plnyYqCdrvD5z/3Od7/gQ9gjKHIMl556WW++pWHWVteQUeyRiIlTOuQx1vl146b8L+976YHm2V9yVO6dabeFIQo50Rpsj6fxhNiZ+dmedf99/Hudz2AVopnn3uGZ599htWVVU8v0OICgzf7VJPM1tuKTNFQJIE4ceI4973zPo4cOczK8hLPPvscZ8+eJs+yycyqEoJwEsxBtRY7sEZK0mjS8lMEzaYQsv/1//t/+sUFqZ985W/I/dxWkqbs27eP4VCmzIX45RiNxyRRg6qquHT5LTbWV2i1WozznAsXLrC+vkG71eEDH/gAP/zxjzh95kwtP5zEyXQe6luhMUna4OChI7x66g22BkMq48jKkvd/4JcoypLvfO/b9FpN5ma6RMp6MFd5lrLUw8Y68sJjUkVBa2aGX/nUJ3n0O99mc32DKi+IPP6UxJJJdZotf8IRHkgibfDJzieAJr57UlUVedA+98B5kqQiD5MktVWSjAAVUx0URZoKXWH69hVA3mtOh2PwdAPlAdh9+/awa88u+sMh/cFAHGgrsdYKBo/WS72IzIe4OC8uLqC1xVQ5h/fvYdfiHM12QrPRpNYMVUFSQ4PHpKbZ+QGTEoxWAHXnAtFX2lTGUyJC4JD5rRCMQwhyVKb07ipV3UioPHtdKVEibbXbjEdjSg/IO2dJvRieUhLU4ljm/MqyJIljirJka6sv2ZCSEipNm9x2xx0cPnqc737v+ywtLXPf/Q9y0613Ejc6lKXh93//97ly+ZK4yzifuSBmpWna5Atf+AIPvvd9PPHkk3zl4a/S39oC47jtjtv51S/8GjfecjM4OHf2HH/4B3/IG6+/RhwpIgyJtE8J0nDWim+Aq4O9V6avg1HkQXXqLKvOduq8aKcukwtcN6VoN5rc84538J4H38383Cw/fe01Hn/sca4tLYkhRRiAD6tP+c/A30c+SImQo8WVlgMHD3LHHbdz+x234Zzlheef59Rrr4mfYZyideTFJ0X+pdFq0my3SFvtGv5otzskScL/+f/+//jFBamXvv0Nsiwj1hEzszMM+tI9ob6IAlRqFWOsYXn5KhvrK6RpyvZgwJunTzMcjXjfg+9n6cpVnn/hedEqShMx6kzSuoME/oJY2LVnD5vbA85fvkKJwmnNyZtuodPt8u1Hv0Ujiei2W/Q6LbTvOkS+Do60GFCOxjnjPGOYi4Ptr33p13nuxRc4deqUBNjKiGZSHJHEEc1GQrvR8EFKbpJGmoLCl1KVx15UzW0qq4qilBIrr92Bk5o5bk1FURZe9D+4xfpMIdYkaeopdJPLETKpsFganhUfRxEHDx5kfn6WtbV1xtkYp2Sws8jFhhvfXQl6X9aKm42AyymRVrQamoXZDnPdNs1mTLPRAEeNDUb+hlIqlOKaOJYmgtbewiyAw0yCkZSYZiKCplRtJBEwo0AelIFkGVwuq4okiT2FoKq7VEprZmZ6rK9vkOdl3eVrNlo1IXc8zoT/1W7Xn1FVFVubfda3tlBxjENTWUfSaPKRj3yU3bv38I1vfJOLl5e44+57ec/7f5lmp8fVpWv80R/9EVcuX/SGI85bQnmMS0fcffc7+M0vf5nVlTX+5j//DW+dPYtzjrTT4d3vey+f/vRnOHb0OJcvX+JP/uiPeebpp7DFoOYi1QEl5KLhHE4B6BKzPNd7KkjV2U7ojiova+2zYKdkJCkENGvFpPfI4cM8+OCDHD9+nIsXLvDDH/yAy5evCAEYfDWzM+SFhgjIJqXR3pTFMjPT4+TJG3jXu+5jZmaGs6dP8/JLL7O2ti5qvI0WzVaTVrtNq9Oh2WrTaEugarYEk/of/i//119skHLOMTMzw3A4ZDgcSmsyijBVibWG0WiAMXJSRsNtNtbX0VpxdWmJ1998g5tuupk0TnnqiSfJxjKaoqOIdqftQVAPinq5j067x+z8IucuXWZ5Y51CKfYeOsT997+bv/7LvyYb9Om2UmZmOjST2MuwepsoFeaULIPRmGE2ZjDOefB9DzIc5Tz17DO0Wi2cMcRoGnFMI45IYk2axjQTMcjUSos0cVNIoeM8o6xK8rLEGN+O16KpVFUCAoedPmRRUaSxlceHrPEs7MlpD/rnul681ykWKrx+eUy33ebo4SO022IRrvwoyXg88iqiYuzo52XEbsiquvRLG02fnVW0GzELMx3SGGLl6LTalFVJt9sDJwHfUKEjS6vVIo5T0rhBkqY1EKvYSaWw1nosrKgzziRORNBQa2KfMSs5cVRlhamKGkcTBYq8vmmME2Jmq9Vic2ODcZbjLGTjMc1U3HzbrS7D0Yj+UOb6ZrxOlSkr+tsDVjY2qBCuXVYURFGC1hGf+NSnmZ2b56uPfI23Llziwfd/kF/55KeYW9zNK6+8yh/90R9wbWnJN3NijyGJ/VdRFuzatYff+K3f5Pbb7uTP/+zP+MmTT5JXJcNszL59+3nooc/x8Y9/nG63x5/92R/zN3/9F4zHQy9COMmDcA5nFVY5dt6Oul4Lzk5KQf+S+h3U1ADepAcz6eQpFDoSDS9rLYcOHeID738/t956GxcunOeJxx/n7NmzYjrqGezTncDpX71Efs2ZqowhSWOOHzvGA/e/myOHj3Dl6hXOnj3L6uo6CkW316PRatJud2m1u6SNlLQhCh3/m//h//gPxp63P2D8ra/R6/akhb0t/KQ0TWWYtSwYjUeMRkO/0CAfj1hdW8VZy4WLF7HAoUOH+MH3fsDW+qYoKnY79Lo9GbPxsrvWSdsZrTh08AjbgzEXryzRH49oz8/xvl/+EH/7t3/H9uY2rThittOg123JbucEBxK7UgFRS+sYZhnbwwHHb7yRm26+ma9//ZtyIbQwpdMophlFpHFMGovofCA1ai3OM61WC4tjMB6RZTl5UVB5e+vA86mMSGOEtngt16IUOJFAlha8nVpogeXrMyb5l8nikhVBmiR0ux1OHDuGUoqla9d8RzT2Mh0TsB0XDCO8HIfPqEDT7fbIxiMUhnYaM9dto5UhjSK67Y7oRWlF7oMK2tBqxbTbLZppg1azTbPZqkdo8IEpPJxzvqzNJvhQJHN9cZzQbDanMgnhORV5Vr+PlMOV51SJWFvaaBBFmn6/T16Ukh0PR8RxQlUZer0ZBsMhg+GQqqyYn59nbnaObJSzvLxMf5yRW4uLIsZjb3HmNM1miy//4/8Va1ub/MVf/RWjUcZHf+VX+NI/+kc02i1eeP55/v3v/i7LSytE0URlVWmFcdK4abZbPPjge/nsZx/i5Zdf4uGHH2Zre5s8L3AObr75Zn77t3+b+991Lz/80Q/4d7/7O6wuL9cbUmg8OCvXKGhueaVEFNb/jA9S/swJeyNoP0l2Fcp0+TvqNYTnNznflQvl4t59e3n3e97DnbffwZUrV/j+93/IxYsXZRg54JG+NNVeFE/uMXzg9n1LpaTLrCN2797Nbbfdxk033Uyz1WFlZY1r15Yw1tJotmg0WjSbDdKkSZIm/NP//l/9g7HnbQepN378fbkh+gOKPCdYVFkrPmqj0ZBxNpQBXOcYj0dsbKxTlgWbW9vs3ruP5559ltXlFbCOPCvo9ro00gbjTF5vPFBXmop2t8u+/Yc4d/Eym9tDKuBDH/sVfvjEE5y/eAHtIFWKxdkO7VbqxzT80lcKjViOV9ZSVJa40eDd730P3/zmNymKws88OWIl0rhNHdNIonoeL/JtXsGWEhrNJsZZBqMR41FGVhYUlak/0xgr+lrhOJTPwlJfGnkgP4C+07yYydWQEqsWIHMSdGIdMTszw403nmQ8HnD5ymVxAlaabJz5kieUBZOhUesxr1rYDCGeFllGp92gmcQ0Ik2sod1s1EJ84/GYcZZ5igMszHXodJo00pSZbo+WH+JWOKqyqMurwFsaj8fkeVavhYDrJEnK7OxsgKJwTtUYnXLC/RmNRpR5SbvdFY2msqLRbOKcYzQaiedgWTIcjXFIhthst1FKsbGxgbWWZqvNnt17ccZy9eoS26MxuRMFSlNZPzkh/oW79+3nN37rN/np66/x7W9/lyzP+fV/9Os89NBD9Hod/vqv/4Y///M/ZzAY4Cw+WHkw219FpTU33Xgj//gf/2MajSZ/8qd/wssvvSR4mdZ02m0++cmP81tf/jIXL13g//s//X/46U9fk8FcT/x0VtVYUOhyypLw19NN8KpwbXfcyL7bWps6+F9rPxY1WXNB+z4gT7t27ebBB9/L3XffxenTZ/jOd77N6uqqp3VM2VXJCqopKDqaKGfIDKuMcaWRKE0cPnqC22+/nQP7DzAcD1ld36AsqloMMkkSvvwv/9k/GHvedpB684ffE0ylFDwlTVPa7RZ5NiYfjxmNR+T5SHbDsqDf3/K7ZE6vN8OZM2/x6k9fFYDcCo0hjkRAL89zsjwnioVVXVQls/ML5JVlZX2LysAD73mQsxcv8sIrL/uunSbRivlem3ZTZtNk8YixodYxDiXBSGne87738vRzz3LlymVJOvyF1ErTiCKacUwzSbwIXoRykyHbOI5JGw0KU7E9HDIeiYZV6ECFu85YW+MAIdA0GnLzOyOZ1DRFASYLHSapvNaRqCkAcRQz2+1x7OhRNjc3ubp0qVZkKPLCC+ZFU+n5RB6mfv86UAi3JokiWs2UVpoI5SDSNJIE4xxZLiqqZVUKkzhWLC506bRSEq2Y6/W8LE2Kc4aykLJOZu8SRtmY0XBEWXlWvoNmIyVJBGyfmenVWJPWMdYLJo7HY7Iso/JyyjgJqGUYg9ERw9FIOqMOiqqUrp21aB0zN7/A5cuXBVBvteh1Zui02wwGA64uX6N0gmc6YsqyoKosKI0BDh85wq994QusrqzwV3/91xRVyRd+7fP8d//4HzMYDPh3v/O7fO973yMvShHMCx17X47iM8qZmRk+//nPc++99/Hss8/y1Ue+ytrqKnmeEycRN990E//yX/1L9uzZx+/9z/+eJ554HAJYbn2u4zwGVO8sTjIlP1qz824N19jDDjipADxApTRT5Vno4LkQ7wjKnNZDNIcOHeYjH/kIRw4f4bXXX+exx37M6upK3S3VSqGdmFpoHYk5qJZMLknEWKPZaNBsNOXZ6tLtzbBnz24OHz3K4q5d4FR9ra21/Ppv/3f/YOx5+2MxjzzsTSOFgCbDr0JsHI8HDEdDmY73TN/t7S0qT1Vw1vHYj37M0NP2ZQC18GCsCL3lRSkeeM0GTkG7O8vy2iZbgzF333MfUZLwrUcfpTAGtCKOYppJRKeZksYanEUh+s8BvzFemfC2O++gPxzy0ssvU09x+65cpOUGbScpjVhE/eJY7+D/CBEtJitLtgZ9RmMxiKiM5OahSzeNzYCqs7A0lSAaAHPrxe6m11odqJxwjJJYxP12LSyye89u1laW2VjfoNFIxQevyMVdxM/vRV4wLixbFfAoH6ks3oXaWhpe0aGZxCSRkFVLI92rylTkWeGtpipajZg9CzM0Y0WqoduR0aBGsyGaU1qxsbHJ3Nw8UZKwtrZOXmREUSRei1FEq9lE4SjynD27d4d6hlanS1mUrG9sMOgPBLeKU6xVjMc5jVbLj4nIMfe3pdxDR1RGsixjHUrF7N27l/MXLlCaila7TbvVkcHXNOXq1SXGVYlTETpJKUuPG3r9JItjz969/OZv/AYAv/8ff5/hcMg//1//Cz79mU9x4dx5fuf/9zu89PIr0ijxJ7XutkkUEOBaOe68806+9KUvkqQJf/zHf8wLL7yAl6xndm6WL/7aF/jwhz7Mo48+yje+8Q1WVlZlGWiht1iH7067AC36BtvE6WeqvQLYAG5RV4H+l1oHaurhCDFQ+RlDWbvKiSbUiRMnec+D72H//n1cvHCeZ559hgsXzoPzTalIaD6iNOs3OU8vaLfatXRRp92j7SWe0zSl0+0xv7DIrl27SBPpzH7oMx//B2PP27a0aiSpjDN4WVlrDKU3AsiLvBb2MraiLAtKr9I4MzPDc88+K3pAcSw3eCadHJfIx5dVKXwaFcusUJp63RzLgUOHOHDoIF95+BGpfZVkGmkqWY9CZHRl6DXgOzF4pvi+/XuZm5vjiSefrC+cv55oJUEqjWOaaUrksaEoEq8e47+DdQ5bSWANz7I0CJvWD8n67MARMrCwhBT1ANV0dsMUwLADicKLrbWYn5ujNzPD0tISg0GftJmSZdLBC5mgMJAhDgQnF3o/rn5fhfYAuqvLQodgHJV1FFXBKCsoreAWeVmS6hRnHamBsrRoD+waM/LmnpWnG1RsbvQhapIX2yytrJA2Upppg8GwINKKYSbZW5lltNqZ16TXELXY2NhiY2OL4TgTxYLxGNBkeUlcyShNWZUkRcVoXMjQMWL/Vfqs1JqKwTjH6YRsXGApsMSUec78rgXiZgs3kJsfbx3mBKKrs96Va9f4/d//fb74hS/yT/7JP+XvH/57/uAP/4hWu80vfeADfO5zn2fp2ipLy8uEVpt1rhYuCdiPw/HKK6/w1rm3+OhHP8wnP/kJ0jThxRdfpKoqNja2+A//4T/y9FPP8hu/+Rvcceed/Pmf/Tkvv/yqbxhE7KQ+hbRNFA3C2nX1Pwk51LkgMM1UuR/WHjtSsJ1BKqwRUEpwwDfeeJOzZ9/iwMF9PPjge/jMZz7NW2+9xQsvvMDy0rW6SxvpiCgWPalmo0Gr2aLZatFsNiXbbqR+pKzh/SihP+hTVhW9bo9eb+a/HnTC3fF2M6mn/uYvpTuUNmS+S2uck66SeKlVFFVBlmeMRmPG2ZhOr0tVVbz0/Atkw4zxUDowlR8yVVrXQKsxVgDSJGJ+9y5GRcX2sOTDH/sEf/nXf8v25rbwVpQjToKeeITGShYVWvpKkySNejj3gfe8h8cef5ztfh9jKrksypsX+lKu12jRajTqIcokidGoWgvJ4qiMY5CNGYxGjLJMJGui2ONXqs6kjA/kyvOM0iT11lZuB1s6qB1MgM3Jwmk1m8zPzdFIUopcuEE6UgyHA7Jx5ndXwTIin1Vqz0iXRWjDKmW67SPEUPG8k6FRBdYKv6uqwJMh87wgbQjg30wieo2ERIGyUyYShcjvbmxvoJUYYKxvbtEfDmh12jjryLMxCuTztHSG9uzZhY5EYbPb6ZIXJdeuLdegcGUs1m8l1hiarRZFKdiODBdLre6czHaKE4ph7969lGXFysoqURzRbLUoy5y00WCmN8vy6qrQRgidTsmCQ/MCPyCepikP/eqvcujgIf7jf/xPlGXF/+5/+6+46+67+PrXvsHff/URtra2qExZl01CEdHgg0UUaX+NDTfddBMf/vBHWFtb5Vvf+hbr6xv1xrawMM/nP/95PvGJT/Kdb3+X//w3/5nVlVUp+2q4028pIaXypaVP3yaggZoEqZCpaqXqgCVMCl1n12G0RlQpJsoG9YbpJtnZkcMH+eAHP8j+/fs595aYmKyvroKDJBUTkna7Q7vdmQSrRlNkmTsdmdRIp1jnSYqkC45f/ydf/gdjz9sOUo//+R+LomMUo/3oRlEU5N5HzVoj7iojoSfkVcn8rkWuXrnC2soqW2ubrK+uk/vWtHXhpraUhXQTkjQhaiTM79rFlWurvOs9H+TNt87z/PMv1e4libeRShKxFBeZVoOyABalYuJEVD7vu/9+VlbXeOWVV/wCMr5uD5xaSHREq9mikaS180kUeY1yHzQqa8iKksF4zDgXmytjHFEs0/mB++IQgwhTWZ84KR8QhAhqjJ0KUtS7XHgtQLPZpNftopUwpqWDpym9nnlgJztUvTDDoq+HkvGL7Tr8IvJBOY4DY12hnKWsCqog3eskkxKpYk0jiUmVk5TbCPtb64iyKGm2Urb6fbrdLlVpWN1Y90qNDcqykFQNR6/TIY1icdNJRfu93x8wOzPLdn/A+taWUDpsOBdacBoH7U671lMvPY3DeZMKacvLOZ1fWCBJE5aWrgkXrNHw2byj15the7tPZSxBKqbeGKyrGwxhbERHEe977/tYWFjkkUceYX5+jn/+L/4FB/Yd4FuPfpuHH/4KWZHX2bCU+xGTgMLk/PpS8stf/i2GwxFf+crfs7R01VcOoh7wS7/0y/zmb32Zsiz5vd/7PV548QWKvJoKFkFwzgeZgCsRpIyo8zkICbvXEnfhKCZIe42DeoGr6yNAOA8oW2NdsY44fPgwDz74APv3H+DNU6c4/eZpsiyj3W7T7kip12g0aDfaMtPb6dBoNYVy4ANU5PFFrcWE40v/3W/9g7Hn7Zmxg2guKwQsLXMGwz7D0QDrRDyuqkrybEw5zhhuizLnaDRiNBrT3+qztrpGWZW+dSk4jpR0BiKFTiKshqTZYpQ5brr5TvqDMT999bWaTJYkYnbZbCQ0GylaB5ARfxlCyeWYXZij2W7z/IsvYH16qrT3hvYEORcsf4yhrArysqSoCuk25hmFNVRKwNGqkiFYKTfCIKXPoEIHbQqMnC6pAkXPEuQzvIYVDqMcFQ60llnIdluGuLOMvCi9C882o1HmMwHq9zJOxMRyW4nDiTXydA7jjVutz6vCrwYREcyKgqwQR57cGMF4qoqiqjBWFBXzqmRc5GTWMLaWQVkyLCpWt/uMjGU7K6mI2B7lXF3fZJRXZKVjVFRklaOwCqdSisrRH+UMRxkbm0PGmaXVnKHfz9nYHjEsDONKkVWOcQV55cgqS1ZZhllFaTWZcYwLSxaMMS1UQIWiQtEfjTEoL8VjyYyhsGLrNCoqCuMoPNBup1r6Bl/O+2flHKUxPPXs0ySNBnffcw/nLlzk9/7D71Nay2d/9SHuufdeVBTwLI/3uXCeg/6Bz/ac4/LVK/zO7/4um9vb/NN/9tvc8857SZstCRjK8d3vfYd//a//R1ZWl/m//ev/kS/++q/Tm5slOFs6X5ZZJZ8l39uJI7Dy4zJKy8+EdQVUDowTjanpc2UQlVvjoFJglMMq518vozayzkL3TuSBlpaW+PuvfpUf/vCHnLjhBj78kQ9z8uRJ4TnGvmOXJvJMwqjXZIxHZg790SmDc6Hx9F9/vG1MqiizeqeWLlXpPwjKsmQw6JOPx2TDIaYUQfxrqyssLy2xurLCaDyW9mUkjq55JUO2lbWoSABN4xwqbtPt7ebo8Zv5u4cflrIQ0VmOfZAKs262MlhMPW+Gg8IajNHcfNttPPbE47Kbxqr2+wJAaWzlOTwKKmtxlUNpUwc9bQ0GWwv85aXgIda6GsC0Vuyilb4OXprCnsRMdSLNUguceXAdJyWhYEWa0Sgj0hHWWypJ8SYjNNpNB/ggUItvEMkAr3aSKQoNIwD7DoWWwFX5NN+FX22tboDx83I+s7PGUjmIIk3hWfbj3FJVloaKSBDqRZEVlMbKDVDJxqOUzC86azFZQeLEQENpRX8wotNoUeQl/VFO5qyX1Y0IIqChnLGlIUZRGUthLTXL2gYMTrLy0o//1A0MD0Ab54iswSgJYBGSeUqppurgrUJF5eQ6bW0P+M73v8snP/FJRlnOCy+8yMNf/Sq//dv/jIc+96tcvnqZc+fPYy1+KNzVpZaaHL78T2m2BwP+4q/+mo985CP86uc+x/79+/nhj37A1tYmGsXFi+f5N//Pf8MXvvglHvrcrzK/uMif/PGfsLy8QpiSlGvoN1ZUnelrFbIlfww1qCXBCahdkHc8ggqrP9/Km+B6uE44VYp6M1a+OXD27FnW19Z41733cf8D97OyssLlS5cpioIkQCBes16yTOdXqpH8UmlfYr6tIu7tBynp1Oh6hk0pibtlnjMeDRkPR5SlzK81Gg2qomBjbZ319XW2B31wos2tlRLrKgelcRRG/Nks0gK3KubWO2/jsccfYzQaUWQ5USycDHEKTlDKiZtsWdTzYBbt/cosndlZtgcDrq0s+wXEFI6smK5wQ2ZSWVsTvUUX22IKkQExRqy5rQ30AVW/T+jkCW7uOS9TJ7+SOwXlg2EV7K+RqXVQZEWJAxppilKKKsupyhJjPSwbdKaRC65rDETVu5TxwUz74KCVJvbW7XUHyOEzTeoFK2s34Hly44ueVwCCYTTOkBEWS1Va4iRhXOSURqymTCU4klh0VZNREr+pRUqBimRjcpZ+v4/rOuI4xSlVu+3itDe4xM/8gfaqjmKAEc6znyObusHKUgxTozjGFOXk2vhMOWA6tWSKP2/hVpk+JyCkynPnL/DEUz/hVx/6HGsb6zzy9a/T7fX40pe+yMc/+Qn+8A//iK2tfl1STQoTWYcB9wnvWZmKR77+Nc6fO8cXv/hFDhw6wFf+7u+4euUKysH29ja/9x/+Z966cI5f/0e/wW//89/mT//0zzh/7oL/LpbpWzssaZ+neNyJ+hhq0ByRM57ep+ujmlS+dXAKo06oiYtQ/a6+NC7ynKefeZrV1RXe8Y53cP/97+LCufMMBgMZSfN+iaGDH/JMCAdp3maI+m8o9yJtiTRoZYm0RSuLKXLGw22G/S2yYZ8yzymNwcWaQX/AqD8gH2f1glFay2ItS7KsJC8MhVVUJBgVU7mIu995HxcuXuDcuXMMh32UlrGKdqtFs9kSEl4lLXhjTO3OWgGVUpRKc+c97+TNs2dF/gNXk86AyeCtFacbp/xCdap+WiNzg0VRkY0LyqKiqizWsOPCy8/KbJyruyjh3yZ0BGMMpQ9Q1nfjdBSjdExRiS610hFWRWSVYVxWZNZJeq40Ff5pRQe7dIrKyp+NVVRecrYyVqb+S0tlZBOwaKxVlJWlqERvOnwXefoOmZOB4CBuFwJyVVVUpRV5GZ+FWY+tyQydlJqh3DRBssZNnpVzFM4IW78oMApGRU5hDURahN2MwxmLM6YWT1LOyZ+t9XK1hPTUi/H5QGTBVvIesYprgbj6PStbZ9sWhXHKazZRK2EaF6R7w7WULvHLL73KM888wz/6R7/Bnj17eORrX+MnTz3Fe9/7Pt73vvfSbDb8ugqW5hrjlGxYNqwpPHG1xFSGl195hX/7b/8tKMVv/tZvctPNN+OUmIwopfnWo4/y73/3f+bWW27lv//f//c88MADootmnQ8UatKlDufEB/e6/Lzuad3P//twbNNdQB+/fTZFnQ0p7WqYI9KKRCtWlq/x+I9/xKWL5zl+4ihHjhym0WzIsfj14MK0hXchMlXlm2cTMvR/7fG2g5SKRXHPWEPhy7utrQ3G4xGDQZ+yEKPPcSZuMlubW4yHQ5y1nsMz8aQvi4o8L8lLI+Ji1lE5xeKefRgLz77wIuMix1rv7eVLqKqqyLLMS8a6+lnVGErB/oMHMX4UxyHsWBFxm9ol611Y6mTrKixSOjonT2PEsLTwGZs1ps6i6ovqgX/nlSiD8gJO45yWRWsVxlB3lCQrkOn30XhMUZUyXBzH5FXQu3JYrTBaYRQea/EYA3JDldZRVRKgrQ2B1d8oxmEqK7OE3jq8muosSglqvMa2ncoIw2KXUs9466pQAoefqRVFQ7APxNfpXdf/fGUNha0obEXpDEY5dJowrkqGRSYq/2pyk4UXh/fRvgNcl6fWMqlcdn5WGKcBvIropLQN+kb1J+z4vjDBTsKClzLHWsN3v/tdzp8/x6c//Wm0UvzFX/wlRVHy0Y99lBMnjgvlpcZxXHjbyZv/nErr2vIy/+k//gEvvPAyX/rSr/O+972PZrNNVUFVwhNPPMnv/M6/o91p8+Uvf5mHHnqImZlefYzq+gg1yQd/foZyfQoVXj1B3WvttPolaufvpUnjDW/932t//U6deoNXX3qFTrvFsWNH6Pa8/pfXNDPGUPnfV36D+4UHqUgrqqogy0YM+ttsrq8zHA4Y5znOlx4jr2e0sS7zOuPhGI0m0qKSWVVWzA2NJTeOwjjy0jDMC7YGfd717nfxk6d/wmAwkIMTAR6f7cg4Q1mWFGUlu6AKQu+SpVTGceL4MV54/llsVUrZE4JcHZSC1MVkUi6471bVJCsoS+uzJN9J8ZQBXzHVNynWSqAwk6BpnfVPh3GTTMc6LdmXs2RFgXWQpg0ZVakk+IfRiHoRyVHX2Inzgc9avK9c4Ot4kN5nCcbLq4TvFSAAx0Tuo17Y9VoMNu5em9t/v0lXIDwlS5DMULKI6TtDKbzkCzu+h2HC0DYKsqoEp4h1QphTlJlLLx+id8qW2KA8UWer+Jk3MazNyxztHbZDuBAzAucnD9hx/UEaCoLN+PaCk5ZD+A8NWZHx8CMPE8WKD3/kQ1y6eJGfPPkUR44c5FOf+BXmZueuS2t8BzZkVHXpJd9QMrWI4Sjn29/+Pl//xrd4z3vfy8c+9jFm5xdQxCginnzyKX733/17xuMxH/7Ih/nil77A3j27iKaON3zWFFzvr+l1T7m8Um1NPTXCStduMjsa+ftdCZzmf070sCbwPDX+piwo61hfX+fll15mY22N/Xv2smthljSJJqYTxmCKUtyGjBVHo7fxeNuYlMkL8iynLItatN74utJaw2g8Yqu/SZyk9Edj1tfXhevh8JPxlV80Ebm1ZKYi8yCyAt7z7vfw3HPPceXqFUAitHUOHUXeOVg+R8DyyW0cbjatImbn5nCV4eqFizJSoiOvhyRlnWQR4cTIonFW5DuMT3Pr+twBVtfrzvnOj6TUAaT0c1VWzkOEktLThqCiwInQmzJTu7cCrWMvdyIyL5PAwXW3987XGefQ0//uwgTWzmNXanIzX49FBIC4Phb1s3vV5Gdk6U7PG4bH9GAxSgaqJ99BrpH2pqNhsNVag9YRwleqSOKIRpyQGUttUaCm7B3UTgxxgsRM/Vkh7jJVRbutCbibU1Ov1+q6o58+hTv/JWxhhKDmB5r//u//nl/7/K/xznvv5bvf+y73vPNO7r3vXZx64zzffPTbZFlWZziT93T1abteogUnWlpPPfUU165d5bMPfYZ2t8t3vvM9lpfXscbx4ouvUpZ/yOd/7XPccsstzM/P8chXH+H1N9/wEiueYFAz4fFd+Mlpmo6fNb1gctl2IGlhPYinozRsIt+smWhPTb3e4Y1QLVhDkY85f/4tBoMtTpw4TrvbYmN7m3E2lsYYwn/DuZ95r//S421nUsOtLfob6ww2NynHGc5U2MpQZjmjwYDNzU22t7cYZxnbW9vkeV5jHFXly7GyIjeWAiicIzeCv8zPL7Jnzx7eeOMNnzUI0KkiRRTJFxE53IoyuI348sk5hbOaoig5cOAgmxtbWANaRf5mwIvdy+ummxzWKRlANpaygtJAYeTXyioK558WCqsonRY8iNDSladxEphKa6mcpQKRKMbV3ZjQ3lVeriRJkglOURpcZcWk0cl81I5n+M/pKXLezhk9WWxTC1D5cVBVIwv1oppsrOG9VU0NCZbrk+ek0xOe+O5ZwIUUkxswHJfzW7VW1O8jC1qyM034vTdfqCf6J98nmvp8Vb/39IR/GCbH6z1BHCfT8bcezg3D2pOH8plA4ARNPad/SqlaiufCxQt87WuP8OCDD2BMxTe/8R2wEZ/85Ce4+647vfXW9Tfe9DmcfvrNNdagNWffOs8f/8mf0Wo3+OxDn+LAgT2goKoKfvrKq/zpH/8JGxvr9NpdvvSFL/L+dz9Ir9UhCt8gBCg3ecYOYid5Wf101z2ByDli54icIwFSBZGzRECMEEIjJWThyDkiZ9HWoE2FMpV0hW2FsyK4GCvDcHudM6dPMR5usWdxjvnZXm3p7ozBVRWquk6S6L/weNtBan35Gusry4z7gj+ZqqLMc7Y2N1lfW2N9bZ08y+sMyFjpiOVVRVZVZGXFuCwYVyVWa6yOsH4q//77H+DJJ39CUYjppnUWFVHzoMSBxM+9+VLEOs849uAnSnPnnXfxxuk30XGM0pEoE1QVRVmS5YWI0hkpCwOwXFTyLH1wKg0SlMwkaBVGApUEKF3zTeqnk6BUWEvhoHIaSyRPpWU8QWvSZpM4SWXaf5x5W69AR8AnBT/vP+pfwwKXp9iiJ1EkQ5+KWqwuqqkHHvQMwUJ5nM7P7ImFGP61/in3DVFE3UbWPu0PcVFrpgJbAFYnn6U0vssY/j3csh4j8iYZtUtMHE0+I7yfmipfmHj8TT/DewfZGNTOQFdX+uEchtfu+Cz1M8/rH0GS5PVTr/P8C8/xwAP38/zzL3H69Ft0e20+/vGPsn//vhoTmy6pdr53+G5GKC8EWZ+UtbVt/uRP/5zhaMBDn/s0h4/uJU6kY3zm9Bn+6i//ku2tLfr9Ph//+K/w0Y99hPm5Gc8sd9IcUFOBqg5OmtgpEv+MoX4mSpFEokqbxlpmObWUdkmkRY5bCXVIOUT5VsqngAsQOk3OVlhTYMoCZ0rGwz6XL11gdWWJmV6XPbt3i5GtlEUibPiLDFJra2v0t7cYDAa1hOvq2jqra+tsbGyzvT2kNA6UpPIWRWEto6KSZ2kYVoZhUTIqKmnNozhx/Ab6/QGXLl2ux2ScX8jWidZzZQT3AVV3MCprpHPku0lz8wssr66ytr6BccGBpCIvJIMLnS3pclny0pIbQ+mcdHYcGKcwaCon3TSDxjj5s/HtcemUTJH48BmVE4cRoRg4v3u7OsBgLcV4TDbKZGwG5d1NINZK5GW13HSRmiyiWClZSBqSSJFGilTLn9NI0Yg0jVjRiCCNoRErmnFU/0wSSRcm1RBHikTr+u9irYiVxyAUnr5giZW4HYcgoJWrn0oL/USCmfVPeV3o+srTEfvXBhsk5SOGsKjF8t1hMLYkjgM5NqyeSXm584Z3O55M/V5rSLzl1/UZEZ4TN0mYptjaanpz+Fk4R7mQpSoiHfP0U8/Qabc5dPAQP/7Rjzj/1lkO7N/Hh37pg8zO9CbvjagS1NhN/Qw57KRLp5WQJrOs5OG/f4QrVy/x2c9+ipM3HKuVF868eYavf/1rJLHmzBtvcMvJG/j0xz/OrsU5H1z8GgqbDpONJ5r6c6QQN2t/neR1sj6iSM6jjlStyR5rCWxhsmJn0ukCOCiwhzUyu1uWQqMpS9ZWVrl4/jxaWXYvzJEmka+yfsGYVH+c4XBUFKikwfZwwNLyOnmRU+QV47yiqRKM1VQuonDCixnmJePCMK6sH4Yt5UZImkRxwj333MMPf/hDmclSITj7IdBqghU4ELeRoAkd9FaUlHCz8/O89topTE1jUhOTSze9LPE/4CZMyPpn8AWNmnxqjY/ITeOmhPSnb4T6BtSTYwakfFMeN9BSOoTjcCEQAPWQMv7GmGz/EyBa4UeSnL9xtP/3UIb5BV/fpBPxu+nFJce787OcsjWovvMLyOc58ERLV4udyXmTzwlfO2DoIbsK52Ansg4OUeGMlJTjOhayblVKZuWmfthNPqj+m53vKL/TQKvZmGwMky8LzvoSK7xC1ZtJeO3Uwf3so/5rx7A/4PHHnuBjH/0YjzzyNU5eOUJ/OODee+/i/LnTPP3Mc4xHI2rJlHAdd5zb6ash68ki2eh4OOAbj3ydT3z8E3zmk5/mm996lFOvn8IZy+k33uCv/+LPeeizn+Xi+XMc2Lefz3/2M3zv+9/n8qVLvolRYwJ1xy6saa3C6lOhFq5LchSyRsMGqiSjiiNQWA+u6zqwKkTz31jEu6+ScVGDo9LyiYU/b8ZYijxj7569zM/22NoW1Yu383jbQWqYl4Air0q2xutsbG6yud2XrMlAaRWRjbDEFEaTW82wsAwKS2EhtxHGKwRgDc3IcvKGG1hZW+Pq1atEkaS1xkgHzzlJW6ZBTWNcjZvKDaACPYxOt8v58+f9P3uipb8Oyu7EMrgOGJVrFnZtV+988ufJYqp33fDG0/edDp+787xJCh4ImFPhT6kat1H+c4IVw4SQNylxpoPRZMH521FNZQb4IDV1qysmi++6MC2/uqmxHTsBxJU/vqBxZFQgbkx99fqekHe0U+cq/C1eD4nwOgfKTuYOrSc2RVrjIuV5bP7o1eQmlzeYPu+T7+crPYS0OrkRpbSbGJ7WQdv/+w78yr/Pz1zEydUQs0uteevsW1y9epWbbrqR115/nWPHjtBIYh589wNcOneOCxcHkidNY2G6fqM6YNb8OiUNIeHRKaqi4ruPfhs+ZPnlD74fbMWl8xcwVcXK0lUe+epX+PSnPs2F82fZvWc3n/rER/n6177O0pWrBLmfcOrrzaLGCOsLgwSpyRoMsV1ryU4FOrA1pqfA1xjCf8c4P5ZjqLAo5bNk5aicQTtDpBway3iQcaXIWVhYZNeu3eRZ9HPP8/WPtx2kVrdGKCUl6HA0ZpiNhKWNkPFMZalUxbXNAZnVZFYxKCzjCimDlMI40Wx2zuHKkoXdu/npqVM4reuOnZAKbS2RE3SyawpAvUh9aeAlevfs2cPpN98kiSNkmFiWonaSuupAxdcTvse0ROoOzCAKYOrk4tWYkB/anOxITBaANyeY+ksp+ZQmXA4fW+t5vxBUFUwEyoCa/l7/vj5QeYWbZAvX54naq446Z3Z08ZR/XZ0H1ZlmoEgonHZYJW3zcKqVwlMHAg1icpShLJc/7IDod/7f3xRu6jNDSNNK+TEX67M0CariBDUV3KYjbMim6qRBsLaqqOogFQaJg8OOhok873QwDefSX6D6zLvpC6LqY3DWkY1GPP3UUzz0qw/xve9/l2ba4M3Xfsq73/1ubrzhOBsry+R5LhsXtr5u4XxO1rYJX8dnM6qW38nHI7776Df5yEc+zMc/+iEee+wxLrx1jsGgz9XLF3nkqw/zyU98nDfePMX+A/v55Mc/wg++/wMuXbzov5MLw6OTtY2b0piSb6qVV+0I5ZwSKED5i6lxvqSXdaUxRA5p5Fhvbmt8FWEcTlus9tx+V8macgYVRZgCtjfWUc6xuLCbt/N420FqeWuAtU483IpCvPb8sG7gfoyHGWM20WlKVlnGpSGvrDBbdbj15Hbq9GaYX1hgc/NZz+ebtCSVIHRycrXz7UqNiiaBRQwJhcHeSBJSDZ1WQqxa9f2slZIorkIbdeIXF4wkZfDY/3wAkaOJ3busy53dL3A7wDwJWhPAdXLPeqa9k7Q5/GPI8mpOlA3piGQBYaA6PGR3n77p9dS/TThPYchaWvgWXOxLGvlZ68RWyvnQGAK+6EL5DMYpP2xr64zGnxzZOJxXDXCTI4yUmkplprfqUANOyqsdGZxXPYh8YAqE0BDsp752/VahhJpKrerfJo2EcZH5jUjVeFAURbXqwKSJsLP8VFPnKVzFCXFSoopcT1Be7G3pykXeOnOam2+6ieWlJfbs3sXa2hp33H47p18/xcbqWs2Xc7haikfWjKPO3qfWklOythxaSvCq5PuPPor5YM77H3yAV2c6vPLyq2xv91ldXuLbj36LT33qEzz/4vO4Awf40C+9j8cee4zLly4JM9/KyQ1Z96T8c4DeuUGDHyOTUi+kl3KNfDCSUkKClnLEkah0RLHy2GNAarVfZxGioaHAqVqkYHt7cyeF5b/yePuZ1LboVhsrWEItd6E1YcLWOSgctDqiy91MEl8VTfGalOQUNx05wsvPPI02Bd1GQqQsvUZMFaWYRkTYsoPJplYyA6SjqE7fQ6Bpt9u0EseRvbuoysJbMQl6qCMPFGpNrMW8UCEZVcimBMCW9xPAUE2vfZ8m77TwJuxSUGcndTDwPyO6QEIW0+He8lP4dvpXP7CmmJAW8Y459fa9g5KkcFYBtmZ8T4fGSZAKWcuEHV7rW7sgLSMjCxawSnBD8ciLsFrUFMLHa6d9PJVSIGRV9dcPN7MPmgGvCiFF+RA1OV2+DFST7+W/rZzHEDzqXHMq8O2IKHJdRB/LEEdTYUxJO91hif083aR7p3Zct1CyB1yursRCAuu/r5QzCmcqnvnJ43zhC1/khRcvcXD/Xl575VXe++73cPOJk7w8GDLOxpKZ+uy5znrD6A07Hz6m+KMLYoqWpx9/jKy/yV133U0jiXj99VNsb22xunyFH//oB7z/A+/l6aeepsj28JEP/xI/efJJzp0+I6a39RnUNXgesrmwQcv31JOAVQepkBT4TVuLO3GsIpnPi0WrPE0Sokih4wilIYp8F7keNtbE3lw3vC7Px7ydx9sfMO5nGFsJE1V7OmPoxARsRWswOSktZnttOonGlJWMjIRFoYQUdteRw/x0sE1v14J8IQW9XgdjKtkBFGgi+bcoQcde2C2OUJ5FriPxn291OuzbPU9kCjTUQvGSFsnNL24s2uM+E8asDrurDjeC/zdbJ/h1Oh5+qcmk4Q6tiVeeh6UCA1w0sPxHyeuNUCwCz6tuEjiH8jiM9RLDNhg7uFCmKb+4QXswOshfyLFNlrebClBhjCUoMtip0tnGDmtjr11v0NZhrJTG1joiJ/pOgkl5hvZUBzZ8hg9Lk6wh/I2TiYGpWDCJaqpeFXXpZpWnE/iMIqhFBGwwNEFCBKrPq3XMdNsUeUHkr1CkNK1228tal6jpqX98hrQzTtZZ1bRlu3xH68erpr4HkA22eOWFZzm4fz9bG+tEyjEabHPjjTdw/vxpjCumro3PwgNYryahN7zvNHl/mhMXKc2Z11/DlDl33X03zTTi3LlzrK+tsnz1Eq+9/DIP3PdOnnvmGXbNzfGe++8nUbB89RplURD594i8gUa97pXaEaS0z6SUj2ah2oh9oJlgfPL7IDqQpik60jLhoZ3XjUpEf8ybyUZx6u3QEpQXanw7j7cdpBZaESDZTBRrCR6xJknj2kABRLOp251h1+5dtdVyNh57LzZRuuz1Zji8ZxaXHRS54FhOXpokdXKmvV1zOBkS8rWko1p+SHsDgqTRoNNtk+7dLSJ4cso943h6SU1awzWW69PPOutAeBzKlzb1q30gqcdT/O5Sl2fO4UxQQQgSICHjkeMBucmVTFvgtPGzfQJYE/Sx0H4BWfm+gZvivEyMY6KBvWMu3n+Ok/MlGRRiQOBEjhlj5Hv5sZoamAeUUWhj0cpgrciCOJ/R2foGUxOnEl+xhOwqNDEmcVtN3ZwTjSIpfSbnRilV9yGUszhPY0A5GdJlKueo2fHTAUTeeb7T5vLWJrETyZZGGtNKI/rZiCSAnHWMUzsTM6bfVtXZs9JK5gBr/FLVn+mcYJOnXnuZ3XvmSZMmR48d5K3zp7nhxAn2H9hLlg0wZTGhPwR80/859JIV4Dwr3vnsqc4flSfSRoprVy7yXDnmgQceYPdcj/PnL7C6usrVCxfYOzfHu++7j1deeYXuLbfy4Lsf5PWfvsLytWWKPPc0E+WD8FRG5TdwlIdBkO8dNNh0FAQVo6lONF7pIKqFFAWOkXs0iiKUjiWT0jKqlMQpkRdN5DrH7v/a420HqTtuPgyoWtM4TROSJKqHY5XHfCTSihf9rl27QCk2NsU5pvIOurOzsygFx08cxFlTp9NBYTLy0VsTTL/DLuoDSS0gJTerimKaiYZGjLW6zofk9gqZjKo377qsckFELpabPWT/TibnCaoF8kFS2/v3xsdKarwFIRwRuFxWZvUs9U1mkWxETUausFJ5TkmUgFJG3kz5Lp0VA4V6/s65OqOCkK+rSdT1v5fpc8DJ7GNk5M5Q1mG1xRlX81WcEy5TmJcMd6+pM0oHWoKGqr+3PCLlsFMhaFIQBxTKZ0lOhRBFcFf232AqCEigr5kk0SSLmn6GIW/nlRaiKKIcDjHjES1fVrRaKVWZkSoL8eSW2MGOr/9y8pvpQKT8rhnGjAIuo/wxg2i/v/XG69z3wP3Mz/e4ePEtrCu55dYb2d5aYbi1RVRnL6FDplB60vX1B+T3F1nDMjXhAXW/np1SmHzEqy8+zzvf8Q5uv+kkS3M9lq+tcvXyJeZ6Xe647TauXF3i8JFD3HbbbXRaTdbX1rxmv/V8vAClUJ/3gMuGcxCwuDqjUpO/V16OJY4j7/oUApQPbDpCqZgkSXwGptFR4pMNfAX2C86kfvlD7xXDhaq6jlUb1aVPAFuVjljctcjM3BzD0ZCkEdWDukBtWd7rNb0tlASdyS4z2W0IN+8O7EAwFmPEsRetaaYJWjUw1nrlhKmLL3dmbSvlVyClsSRpIu/lpWblJV43yn8n8MEQtYMhHnZ3h6lxH+3T4cqz5Kf1rhyiLmBNhXPGC7O5ukwMMjLGTrpySkmXzhpXt8cnHc5pkN0PXNSZgQDhVVWBAmuMNyYVeRvnu6giu2Jr92gpDeOJWoKXOq47rEwCagiY+OwzlLcCgOt6HKg+b0bOrWOCYYboMLlZJkFWK8E/agwU/HcMyggW54e22+02iXLsmungEHkfrRVZaUmbDXlLO7F9x2cL0+x0fJaBEshA+1k157XRlVcAqNMiHeBncPmQ9aUrHD16mCP7DzEeDDly7DCrK8dYW7qMmLxL9j3BfibkSPyad1ObtZRfIQOTIDJRgVUsXb3EkSPHOHnDcXbvWmRrc5urS1e5+ZZbOHnDCVbXVtm3dy833Xwz15aW6G9vicOPosacQukXTkKY46x5cGGCoA5g/j71Y2f1uSQEIo9beYlgmWrw8WJCsJL3fZtU8rcdpBb37BLX3rLyeEboFMlXsj5oOOeYmekxv7DAxtYmeV7UXyKOtCgWVJWkg+Em9/eZq4PE9DxY+IxQcCiUmXBs6s93hrgRg1fylJtepuYVeMkYEdhrtdpiYpnnHDt2nPW1dbY2t2p54nE2kgDjJoG1lmFRARdxNblx0sUzIvQVR5RVRVWWTO4tXwoafyMTaBIumA9jJywjpGNqcZipGzpc45DtMBVYHDXnxj+sD8oye+b1wZ2MFtWKDUEtNGBgJgQb/76hzPXX2zjrg1wIkMrvIhI0qrIkTVP27t3HlatXyfNCNjI/yiQNAX/VbLiy0zgRhO5TUCHVYfetMyrJMMMZMaZicXEXw+GA2V4LpZTXQzd0O20U1NK2cRT7Q550QoOyZijvZCxITySn3VSVoCJPyJWDdTjpX8UxcRpjxxknjh7l4pWLdNptTpw4Tq8ZkShLoiM00STg1pdR1d8bqDtscp0ntIBJdiOBylgY50MOLBxgZrbH4q5FtrYHjLMxvW6Pg/v3MxwOmV9Y4MYbb2Rzc5PB9hY4M3U+Ldfxj+XaTHX3qJe+YsI3i3yADeSaSbc8VOYilyPzlAHLClkp+meHlf9Lj/8G+WAxOUgbiS9LJuWS7KIi09rtdtm1axdLV5foD4Z+kUU4ZxgOC7TSpI2UUBY5O9FainSEtRXWBzv5YrbGpYJolnOOOBYgz1pRUijKEh1FGOedgktTl3jaSbcrjcUEwNgho9EYax2nTr3hGf2GLPeddh9EpuAUwU9CDTKNRTHh1XQ7bZIkqc0nQ4dV7s2JBhMu1PvK32tSx9WsbX+VtY7RKvE7ntwoOwp5F7IbRzAumH7UwL1SXmisrDWxlC+frRFZmUCgraVdw5u5SUCSmUwzcVupAy0E04WqLGm1m8zMzIDzQmeelDths/vgZMX0wk1tRqEDWFOLnJtw2vT0zarr8iN0kPJcXGoCplKfTX9DlGU5CT5QB/rwa8iCZfxHoIa6u6wnumZqx+tCmQs6Ssj62+zbt59ms0lVFSwuzmPzAdoZmXtD18FZ1f8LbzfJaOqSy3/OdLYTmkJBImlra5NOr8fM7CyNVpvxcISONO1GkyRJybOcNIpYWFgkiWKKYlyXssrvgNcHjBCklN8TpNwNJGgl3XY9QdRCVqw9v6ouj8P3CtmhX/dOTQi2/9DjbQepVqvtz5EwTIKaY1h21loajYSZmRk2t7YY52IQGWQylFMkaSq7ralIUkH6W+02RVGgdSQ8rDwTpxFCKSGlUJzEdHo9xuOsnvkJNAitlS9BHFGckKQNCpX745XPltkujTElzliStFEvDGmbClhtrTBmqe9Rv+v7izDNgJ++sN1ul2PHjuGc4/TpMwQ3nOkOWDhPEDqQEuwmduShS6frmyFgBAB6qjx5u49JsIcizydBKnT3XCj5TN0qDpQGKaeYytTk72raBKEEZaKS6azPRCIOHNhfZ9zGq3VOl1eNRoMojsnzvN7s5PuH/0+AY9iZUU0/oyjaKaI2VQ7jkBLE4W3PxdYrbEKTEBOOSqEC6ddJxmZ9Y8WpSSOlzih8hqz8MVf5kMH2BlEMeT6mnSY0m01Mkfk6WfkS3kMJfryrzs7qEyAHF7JHM5VJA7iAASFuRuPxmIXFPaRpkzRJ2N7aYsA28/PzYC0bG5v0ej3a7Q7WlAyHAyojGNV0F7F++CDllPAGlRaLeh1oEWpStl7/UFoR6chjTq6+T5WOJyW04m2v5bdvDtrw7hbX7axSosnN1O22GY3HjMdjWq0WWZZJCWilSxUnEdZCVVaMi4w0ScnyXHZIM+HwxL7LJxCM+Ng1my16vRmKovKXSi6uyIT7sQ1/XHGcELUm+EzohEkwEA2eoOUU0tBQ/0dRRAiMoZib4GL+1+mxEQRLGo/HvPHmm5RlSXndTFK9k6CIE10vvMmtESEdx3B7hp3Hp8hy4uv3CRSLgBH9Q5e63enQbjVZW1/zWSoE0DJgT0qJJbzxZgZhl62zLWsEYzIBp5qco/rEEALIJJBr33Aw/vWE91aKmd4MZWUYjcfUAY+dHcfp9DDgMvU69MRUZx2tdpDxnQSoesxyghzJz/hNIfz8tMGnVtpjJRqMD85hnFyFayP3sHNSok+aKgIWO1OwMDfDwPsC9LpdspEiCAjW113J+InzG6z1GXFYFWEFKv+Z+KOcECl8xm01cZIQaU2R57SaTWIdY62lKAs6nQ65t7KfnenS6QhuJ2KIIkAXMqv6dIUgFcjNWmO1m3ADQ7k39XRuct1CSTcFiEzWtJoKyG/j8baDVOLBcudUTfgLGXUca5rNJgBlUZGkjfpGDGJuLpx352g0JUV31pGmkn5bvwiUC6ClpiorqqqUCxFFZHlBI20RekehRAjYAHWw0bUYV13+K8AaqiqA/oKTheAxXaPLB/rWt53s62FOdlIJTemlOyduJZGm2W5NZVy+AeBT9/rv/Y0dvkXAR3S9y0yzgUNpKK+NfcdEwH08t2WScU0/lHY0Gw3QEY1mWz7NOi/RomscUa5jLG7ISUISxTJL5xxlWZJlWa30aSszVe6Ghe1ATXblurmgbP0962rZn/PCOJyKaLV71AWfc6AmpXT4TtM3wqTMtf4chUumAzxW30z1eXCqvr51gArXcKquDyWm8ms9rIdwHergETaxkGGFj3Na8FFnxFgDRaPRkhAZpJhxP3OpQmlXk23D954KVtff2dq7HTsn5WlVlCjkmjZaDVHCrCq0hnanyXA4ZJxlNBsp3W7XwxKFF6GbrhpcDZj73cqvM4dT0qSZtvjQvoxzPrsKZbH/YkRxDIRMWsrmeGrD+YcebztIBSheKYHKQo0eRzGxNw3N85w4SsGJpz2JcCFkBMSfaueIopg4TSaKnQ60LzG0J4fFUUQUVUQefJb1ZYmaHqibnqyPfBT3OtbGWp+96fq1CgVOHCzqrxSwDqZxBj+jpCYLZ8cGN3XT1NlkAJLr4FlfV7+aJ+XaVMJe/3sA6NXUZ9SlnZoKXv7nozhGoYSfFlJq1I7AMXlIqWWtpdPp+t1yukMYsuMJwTVYEgl/SYTQlFK1RjrW77pTGdTEd2XynuEsSCdpoi7h/KiNq8uB6ddc9/QOMfV5qw/bB5vrOmQ4V5fa0/fAz7sfwgYxCbaBYGJrrGX6lbVChr+4AfeSkg/fetdesljRiGISpQSPwqHSyTFNZ+rhWLVfr+56fMjtxIx2dDqnzpYkP7oeQWv5zVIpLTbnWlMUOZWztNotEptSFF46KARHB6G5IXiU3xx84yOoj0xGwfRkwsMFntfUsSs87GP9PR4oCOrnX5Sf83jbQUrp8KOeXO9LkTgRIldZlhSVxRB5sqNoVQfmsvU7lLUiCeyc9iMZTEoZNNZCVhgh84VZMrsTWPbxm0kbVdVzfRPLKU0YJqbuQoaT6eqgG9J3twOc1D5L1NevUx8oQoYWVqu/Wa7b7Org6P+gpuKq3F/+tQhg72oMKuAvUzdo2N2Vw+Etg7zCZ57n/th27tDOM8xxGqUFh5EMRzKekAGG43F+EVZl4RMzV2/6Cs9rCuB1yDrCZ4XMqf5+1u/ETLLI+rzIpmJ8IAv5SY1dTJ/FOnqr+jyEvGwyWxeA8MlLAnXkv/pQauryqolhyuRsTtJm/5ORmgQq7cddpCMYE3nmNSivBuuNLIxFpRohaFIPz4e3DRmUJZT0U18mnIapc4e/HhOEbHJ+wuu1UsJR8pMaWmt0HJGUDayVjD9WMS7SmKryg+8unDzA1QC4XGtJDmzQ/Ecy7ySJayPQ8SjzHeRwXaI6wXeI0mdNrQhZ19t4vP1yL21O/UnVQUopMWYsKhOciLA+wk/cV92O0sYRY53GOu3vzukgIr/acO8z2W2FRDm9symPOyif5opUsZqKBM4PeO54hB14KpoLmCcXui4RvdRhWLThpdO7v/yd+5m/q+9KL3ugf+YYpoqZqUU5KUU8fvFzFqy1jtKJrnqapjSbTcbjsV8g08c1uZFDsVXzqsLgq19xkwUfftwvVLj+X/yi8yXqdf80udGnUkP3M+/u32cqO5j68/RPTyN34TWClV23E+/IPpw/xqn3vu73Cgkuyk14R1Nfr5ZNmf5Lhbj/1lmujrxGmEZFCVGaEjfEStwZS1mIL0AVysSwlupr6urfK+VbUgFIqwPzjm8/lVGpqWOeZNtK65oJHjqnUSyVjnOWqEqwTrrX1npxwbBO3HWnMQyKqzCO7ZMBJytDJzFpq0Wr3RSIJU4YDPo4E4a4o4m+mi/zQsn8dgMU/DcEKZ0kO/5cD9E6R1mVFFVZe8sZv8ClFJqa5vdPY8VJV7g6OxfO5CdDGRIAcQ9v/pyVp5widmJo4MLArZUfmgCf0xf4usXqy6o6i3E+fXc/C/A5prIGOcif/ffrT54K4w8/m98GmoS6Lg2bMH+ZCpR+EYXOXGlxoyFaa4qy9N2tnw0HO0teOcowbBx+IATiyVFOAu90oApSWvK15OL9bIkZyoKpnX/HDRW+S7g+qj6G6ZgccoWfh8e48L+AX12XpU4f747XTd+FxtXTA/UrfCbiAMIy8t85ZHDG4rtXnhXuh2bReL6gyMV4CXfxIaznVycLWPl1Eb75jlDupn/2Zx91wkWQf1EyJhZFJElMFMV+3EX+zdgJrSdRsWCorkL7g7TGc/II19ehrDDTlQplmgImAUscvCPShiQwcSI0IOtEey6k59NbTr3x/Zw18196vP0gFU8iX0iDnUMCjXLesUcibRBpEAEstaOsAN+GDxyiqb+/fpGGnX7S7nb1D05/RS2RYwexESfllwjthRtm53eqb16f0oabtpbxcGp6NdTHG47j55Lgpp7hqyitJp2tyderca1wLDuPTc6Gmrppwu/DDV9ZQzneqRP9M3yXqTs2fLaU4Oa/sFCmdwE3OdcqBOjJ+0x30n4miLjJ+wRS7s5fd37mDhC+fkzL1bif+X4TqsR1G8WUBlVYU+E7hNfXR2fdjvdWfqBNKXZcR3m9mwrhgdBswfoAZSyVrYTpX5YTCV0jNUXIwpXPUCZ9Oj21+/oDfZuPsL5kRk7LDJ3S/j6AALE4Pz1gK1kvURRhjCHPxt5b0c9Kct36JNw3fnIibKbeIGQ8GhF5ocJsPKbMC5wzfn1IJhfO68TRWcHbKcf9478BOE+nfu9QQU5ESfmGsqIpo5U3SHDosFv7kYvrAUA5EdKmnOwL4X+Tn9We9evCrnkdp1GhcGGMIWxfhBvUTn7/c/MivzinMqnJQg0diMmx/bwgZa973/9/e9e27LauIwF5///vzqmaLGEegAYaICXZ2akzeTArWbYl3gk2LgRJtm1o1A/5lE+K/7Y2yTTfNcMiAalobV1o0kPjV6U2pctIbIWporAk3+vOQkXVhVfmKg6M8Mk1OZ9lnEui4u+gicasUn3WbfoK+zYs7aEfnF5FROh266xTfK/tTzUWoJPX8ZKXwhcoqvDj/XH+uNtMupqolnramB6+WMatPkCmvW51coDPtXJA1TypwFW5OCoowOnX//5yxvTrR45D5efX4Vuk4nqp4kTl4CBqcqqmCw+OrvZXvpDyn//5j58tZ+YnfPz8uDp5SjjFaiwIHEkizDjfCe8bzl911KcLTSa/5Ef05yf36Zi4yPl6hRd53vorcsC3aFYOE3IEM5NTf7VJzNy2SVKwj8G+hY62mrwJLo1hlU8OxGJ/XvY2tFep7hOkfsKpdfXalUXJw24DiHXnRhLIuBcTnY8RYdDkvuPgV8o7YK8KbOdoNT4mbVP1AN4oCBrXAlI7p9erz126d9SBqzhLfwgNu+7LXfOAKka8IpwUsfKZ4/CKy77UpQtT95I3sbhlxcTM/e9Ey6HVu7hxrZRMcSQN99URe+Lwm9+9Xke+832GkPo07OE4OBLnlrlp4J9//slFlOo7rPRibhytLiIi+k9JSedPgFGom+fpc19U664+9XqBg59mm1m/D2+D1OsgdS9Ul39U5Zf5QNj5T/hQ+EZS1dOvrorK2nk2Qp5j0wLAJVYUFpAaaWBTMjrW5Njke05pKtCn+0l56GY9zcWu1LGtiPcYDnj46oI8cUvkkSIJrmTaD9fVc97oySDFXuHpmyYir5BWzY5xGQBUDSqTQAqnVTpT6uoTpMqYEskcMvUDSPEkm2l2wDv75S7OJUg14VRbuxshhqptw6N6Oi8WI+OVTtprJz5JX1Cb0ik02g6bjYGhOt27pFp+cqjo0mehlqIuabtDPwe9qkoduhgSNM6DE3EaPafGkYJfsuiS5qMHsdAk9CEisXPh1bQQ0Dza8Gp9dh/eV/copJguzl0O8xP48gD/g7izhdoHIEn9f583jKBmInK8SipCQ9EZXZSir6ECqPgkY+ChiauRzkZlaMvXReO9TQAhEfEjWum9xQAAlNzf6Mx6iZKxkwBnKQrSkhI4mCzxYeuBv9WJWWnVb+7rkvAS9dkMQtKUZTvxuM9papPspbfZlsn9WRrctXv3juMwqE3VsYHkJm188bHIyaSkyq7g5IKASzRmIdHGGWuKfExE5CxJxieJqJ4hbZUk5MwkwCF9iL3co2pzSRtsVvg5z2K0Mf6GHRbnj+CGH7TS0DMq7riZx1PfIEfStra61muMJwC2S7HldlDg+k74wCbFGZqwM+WhKmdMOBw/Yvx/k92O+B4rHRN8qOoZDsG5CswVewZLMibGURS/18C7rWNdgMS0XbgEAvsRqZobKeIu6PhxS0gSl064E0PPA33HasRtqed9STfq2hHe6tvJTmnv8kAtnhS/VNentL+Lx+UdxSwYhnO6Mu9pZIcrn3obQv7KvFTL3HBZd+Q4RH+CzFb/qzBdSczcNmZxVli2H3lTXqfeL1Dc13+tW5f2+Cv6RcO+9V74QJLiiQ/SOUSga0upGM7ErXb+0//MYqMO/FYgUMADBTu8nWIhKMwo5p2Zj2l7hWW6QXgqIuFvAyM5i7pK4JSfN5O06mfFGbk1SxLuS+dkInHl0FrN+K5N/d4FSGj3kFhSNTSNnSqv1KuQSvj3Thoqg+7abqanwT6bFLfrXjyL+2kpfXZOg6sQN+JnOHOSn1bWw1CvuAUnzlY37AWNzqn2ifBevmxDdOYEuAn65yRxMz+bDP1zusSs8Q7j+Qz7nwVmOI2kji4yYEP0p/P9bZCyNmw5XUUUR+9CxrQiMMRkuwl+i3fkMQnzuSJvhnc6YhB6dp6tZeUje6hDUI6JSPqXbOodCPmO7YUeRNIJ9kV4ZSdaCvQqWf3G5w1MerG3DQZAEU3gMwpUYUmzS51Iq61d1vO5rNt8B1WfVVVGymCsgucYKx+L3s4xaDAm5phai2MBKjAau+tMnBgRRmq2XyWT2DQv8e+B1vk8dK+xxblslbiEhZ/LKWHZTcREdmUvaJrqBdEyMeI/FD5Q9yTpTuJA+zy+jKQZWPxhd7EBUIsRnBr6iLA74QjSkO2N5Y+hAdFzHebWl/lOVNxWo1pbDZYyR9uRfpS9VS2XyltKMmI16SNHamAxjtv2jdW+TeWvAVDg/GlZVPYJ1ai+YOwrziEhEFBLugSIT7YBVT1EKsteIMcpCTHPut+0SVI58u+qImoWfXQQg5BkzBZXjefxNmdtsEZngN2z6wpWEkuKumezYPbMPL0K1iPBb1DlcfxVJGyNF4UnKTFdElhr9VW1mZLHvPDqvD9ZP5CkpAjLQD7EtZLr3nTuACjk+wm2bONS/3Q7wn3O0xcWdbxcbZPn+pZJ0jax15R3RuP7AICCQyMGPlZuNiOhI+11Gz6LkwCFZzGZo0GSCxghVXVK6bTjSfzYGpco4BhMZQewKBn2E5YV4E7g3AJ8qzT7q08yTmbtOTskaB7nQ8f6OBd2GnKEyDx8IaJcA1SmBNv74DFYfVnKb1Fqo+AjAxZSyW8LJqGCuo8mkdj6sP1+stVx+EiSMrPUu+lxAdZHcPMXh/f77yHcQnbGqHKHtDPFbqXnLdiMEPzEtrGWct+pG2XSwW4FqB4JaqbTB+Q0zZM3O4PIsgl8UmC6MCH0RE/TbKXTLoXyw97ONc+qQx1JZH0IpVan27Apw2UB7xPpOQ6tzB5lcRVXJ/b/1vwM+M8FiXiqv1eDj10QunOkJGHg7TvhU8PZ2/aqPxz+3ZBOBWet/2P+14zoudwLo/LbZT/EafnSBL/VKqj+ANGmznL+SiqDSAcQuaY0n4sf0tf4nH54PeZK71eLQ6E/dVkaK2y8uspC3EPV751dYzu+zZGOjv+DdqLHoNghMRjCb9Tho9U9iMe4cMg5YTh16UErYczR/LdhJC4oeJ6hwwY5aaJx8qX6q7zHqMp8p02o3S7NtqbF1i/Kuir3Ci7ehQ6q40KEa7p0FkQ6LQC5F+W5JujpJxEdY4SVvgumwkhDnN2W7ihVLUu2NfkFJbUWXIf9e718z2KO1W/bxcaOB00JMiNGQyaWWUhS74lSBJMjvnEdXYR9yPAPhsRCoh4aDuXfH4T3Pc7lJTCWu1gZZ2HnSXxwDjRxs6df4OM96mBj9zOkoYzHNcEZ0K2zbU2jWhYD8ga5bdMhFw6Nl8FKNh91aZAcdUq3nUXihJ2AnV6Ry47zXMsQHv/VmpHcNNm0ZNlgLF1+2cy2ZowvttCBSvNprVbBIrauGq4jUum9TLfV8AplB9YhdTUpjNSqORbmvWJiQrddUJ8E1egs5cLRNjEUQIW+Ar2GlnG6bFNM0Dix4J5Em/neBEVZyRDwnXoif5zUJ+tlnP/K9WdXN/qLlte73y/rI3VPU3R9iRnOGkfFOp8zSgPbylOfsFoHnwv2f+qOZisolLPaeyBlxLp2KuV2EBVpRd6iqttAfaZCF37++5A5v53lBRtXWnliqeBdgeUqlJD8XDPFWN3EEd5lfx2nZXoBUhlbn2loV8q9MvoHaeZWnjwv3/378v+74QOQOvKv6SmaGyZFVP02UtOfmhmYe4pLD3jj73Xn8VYJJQnArwe2fDfjF4FBghMR8U2Vu+D516Zp32w8pZSrYS6HURPLrS/xZtRn20hJ2TjuLWtNjULrpND3ibvxLjXpJwlQv9pMYcKnCpQEovVaYm0O9w7elv9GPaEejDcTHIoW+HdP5Vm9lgo0JvdGDSdQrbVG5zENgqSLBhUaxJLVqz0r4FeZm72zYZf1wbOrkbiKv2R8m3bHj9q806B7lU3Mfx/ed0HY2ZPCOFZ7m9Sd0P0aVvGbav3ePPk5/eznVAtmVlrDpBI7x+lEPCGP5KMILqEhO41Oa7g5/Q/bVVSkicdzP9na3XMLQRzGpxtVDb93tH4TaqzLGvSR0RO2iCaJ0uRu2U0SNJp1KL0mAu7Iu7EufyQvKP0FctkwzkwpF6p9Ww2MP90SEP1HapyNJaZiLnycbdDTRRsTkIDfeYLqLi539tor3XFyILbt1OsaFzCdKV1tnTGbZf7JH1C3X72kovLGAAiounDx78Nv+ElFOA4R+wnb7OH/k8PQOePgzLiE4Iy15zEeaLwfhW8xQXmHfm036HXB3rgCpMt9RCPo8h4AZxdEB2kDcfycoDx+JTJ91vWZaPcqVpao6jedRPzryU95JZFMuHCJVpWZPBOkyaxb0aFKHVS2BxEHjyBlO8b0upNQeoAkZdSmAiaCtazq2DhOZSoPTCanqZYAEzs/G+3UNVhS1ljBQk3tS1XaorIyhwKWDjIbyBrx53isYaW1OO0i7/NCuYgv+QzawJ4Vb4CK6BwSFNuES1gYzPpf2r4+AykUWqUXeh4+YV24iGNTREXieBB/brnxdAnE7WBw6/F8pzYbVZ2JHC19rhKmdLcpaifFtWf7gRMh24zW5FlNku8GlSbiD3os6TBI9tzXitPXtO6ypoiE86NeCGZZ6D7fTAtKJenErAOVgeeObG5Vqf7c6d/6ZBll13M4jN6VJ0ETkKInUwumiGeH067k5ts6XcOHPyg1rrPym1BOsfMQPaYPF9fjrLLT4bU1hr7pkJb2E76rv5ym0/FM6/PjDAGjB+zeSKExsupVrrl2qvcg2Pi2rN8M7x96pyVBOPc5BasTEjapRG99CZ3uFhPnjLGxPGu+5+9xU7kwWWxExLfSTjFtUu1WVd3MxujsZCLJYY0iELHKWKVKgmD7x3vKzYa8siwbEVOhZfXz0OU+uZl5Mjc6QrfiaEpSa5WJAxr3Bb8LCXlhMnhGn5s+wemUIOb0Pp9HySqYTfntZ/39ZcbzyYMp04FKpd751zqWhVWSAilmahQnJyxNzdZVAGeA1v64ag98eOTYkNv2A3r5LqHZNYmZxAVIJGVb0XONajG8kk5FsLd0rsVqazfxpwBtbEuqrXADuB7D+6D1AUgRAcIgm4PYvXxNXwFcIi4mH9H11kFiU0Yi9W4uCHXsuFyhf2oB2MLOCT95YopIuVHQIxt71XYVe8MXZR9F27ece0xto9x5pMdexMWL2f6ScO5IhBQ5rt343JRFEtQCYwBbtKXVpreB5IbVLIRz5yl+SkDawW5xJlSii6VuHaTSJqUlyYABpDTN7QI/TOl+f1x2a8psv7I63fthN2Ap3UA0NnoRHTdlpHJp6SPE9DfnRVIBo3PWHZt80GeXzZ01eTfiJ+qeCTg8O575FD7iVi2gN5lawTnZuLih8+qI58rvxNv+Wci+9c9ovZ7TQdYhlRW4FuT0dPcnBchbY6KXX4YqRiK35d7JBV77BOX0nEnLWkMVtPKzwgRVDZtgbYJpiRcD8W6Albr8GqiKuUyDNMQGzpZGudmQ0CiaVEwzFIfphipbdtSY1EzTeF60iD6bAFD9vOdTvc+MvqMqWxpuXYzFm3jFgMktaiAzmEnDwZWanRTWcYV5BduulUCrmsi/37/KCuF9F4TYnp7kGTKgWZwmpZpEoqFmgRtZisMM+BPJ16HY7cdngPL8usOdqtbK3UD+q5DcKuP24+JsEH7fmBz1/hM+To3A2bmwLDPmyr8UaXi11az3oIoY2xqUJlioKPhZTVOSlt1W4ddtw3HwFWOJOp4FZHQNPPqMgaA1UzTPTFdVt+OsneHwn2YGMIsJvkoi6INzLktM3Ob4LMlrA5oAKnOGbXrEKQYDqMRcfcXls1u7k4e9Y6y3IzeI50r4qJeWPKTGdCM5Xnugp7zUe/lUXOW2q1+B7DSKQ/FjyRgsu60qow8L4Vt9n8Jne/dUY3e6FrJu9VGVOnMH12vHgErRWCM1A1kGhJn4LcYUslOaTk1SU6p6MQC6g76Lpi2/SlpgPuDze5/nVUmGBj3VYasTdgnP71LTrJexPWeoIEwopwSDyNU9EE1Pk2elmNbirBThd3cA9FPUh8pjq94ySY51VEDIZQfCuIK4XjmWndYZtCYPH0FXpsbfj4uLK/nMLjBYp8/aMN2k++MQtbNtsM29rlyd0TGwbxmWYrIv8GNSadTHXp2VOnJxs/PbHItGXR0DIwJoqeq3BRhiOpslE/rU8ew5fGCTeglWVxK5XYRw505Vyeup1ESOQ+wklSH+580VlyVR3ktf0JK4WRI2+zcVsyU036xe7Eptv2B4HGeBg19Wv6g82R4cQK/cGtCc2dhun6i8wGOLM0Fl0hiPBVQpG/gOJf0z4KrE+KhPsqbCSVHz2Y39Kja8vccEYa6qxEzi/XlwW5PjBE7uO623C5LAvSRV0lJ7SBmRCwKDmKTMGmoUlJvC8Sn154EsxDg6uK/1TCYixAdv5zLZgkliMyl6OiLjh2yizqPIQZOL9QDPdDWW1OiWwNCFqH0f7MLHklRbBp4TK68UUdFTZXLflHBsU71GdeEPNcDFn7mD58HEFYBkYilpJBcWXXtXZKg2u8oApBgtrX0gn3ekNV1AaK1Py6Vxc1LNjEAmuHNKarSwAZBH08timLxlJROoMuNM7FJZtFYZqT0rb22Ht1yoG0Lv/XcCUonGWQ9Oyz9Zsn20Z5LExE6dqcYMd5Ysw/h7aA92NGabSznod4PtDJKmyPGCgqizw6qK1lXlwUN6aAJtSZFGpJ2yxKDSTDoAioUDzT+csCoOACw86Hnea3TvAZTIJ5JUZGoHi33xJu1PFF81JBjfQ9Q4ieqCb0Uz3rsLFxcJDqbUkZYcN8HGjuAcSvizASmRVFt3ra00Z9XrRmp6miBXEsFbgQY9iSo0Mnj1RyVSmtqBr17kw+WoiByvI4atpKk8JynAvTzPZRl75GVmZFOKZ5t+wqRlwFJRx1pazcwVN5M+ybIuzFBG/2lJHZZpyO5HnyumrhIRr1aaajDPsgsa9Q1kWzNcJ7VjjADADicpdMgqE0CqyzYmomHMapzn2VKzpxpY4XvpeK3DbeaHsZpsrzGBUdg9z27hIxeE5NiocxCHiV8eOd0A/NYSkdN+NhW9LEmu/HGOcOZko+Ah8NNQsQPODiqQrlTUvdw34bjZNsPtLlnBYo7O65reWbG4U3Ezo226crco4kjP7gR8diAcq1y7YhbZHu/UufkRL81EcSWZWeEA/oPeZnFGNEMTv0CIyow/pYrWJO72IlS+UudtYucZV03VaQhcgHFa7f911JHk1vhd0lB+3/jK8KRWcYdmB1sN6aaZopfgmDNofkbgn4EWaHEd8OD9cCoZw7VYbw7dnrf0+owXbSHAanzmuPQVvQFUe/H6Mnyg7v2Tw6R6ypm3w5h7l5/mHaaSqp6Hw4Hk0V2aKf+IDhhdCE4VkjTO6ql4xR1rJUJFdd9MxezYBtgcmHT9/7lZRr3ucxMxkwN7Cm98qoZlhkKkxRJ3cHKVuAlW4eIR3Ltk/Ms6aaFdbwd9K/uICqRhSWVkiGPUAVspePyeKhw+G4QlcF0cmcICyeuVNDZloYKYYqCtPsuz3insdlGLQNV2z/MQFbKJkdSXamvAyZXUbTua71N9bZSJuIMlTkAVMaXFLVSHLukD2d8BVNaHYqWURXmgqT2HMT8GgC1i3EP4TJIynFE9apecRKIB6mJ6EzieFJ7OY0wsb1XlOgD9a3+fiqj770CaS1JXeG1cl3ytpq0cH89f4+H9qaFGRKrCZ073WCVpjUVyKtqlmpRgzcI2l+TvcW5qM6vWJjqe5UQq0f7QuOVWgwJCgltX/O76dP8+x6lJUmU4v12UGPN2P2KbsjdqXpcEKN/m9mIpPZWUF1tSdxKsithJ0IYN+Bdt2ZMSEKlP9ANCXbr++KtDNiuJ1bOZnkEs2zbK5Ucz7T5o/wSajYGauHUXfusGYy+UbCEjJGgrEc5Thax1oUhq8ku0eEecO5eO/WiWns/0Wi7wWTkAl0LbgJJtGP71euu+HzzwyXe7sqoO4KIMj001M8mT9NjZsvqkvnkOe8Cyp6oI22BM7EivndaW8n972lW/ayumnQpUrb6lCVFp9ZJLH9E83Kjdy1h3Zua/6SLXFoqBwv5m2KqSkq2k0VvCRoU4qlrxZb35l0sR6cBRCzebNCT55Taq0cz8fY70jBkQLjZS/pyDvCCyCwHbs4jLst8JvwVSRbLdL4UN43IconFRYTD4WwnPl9Q7Nz5k2BcUZWiqfo0ThkR1ZSPactSHmF19DiJcJPBbWU0uRfb21JbXXY8v9sPqVPXPKbWkg9HZ9/iUGqoMfwzLiS+hC0nRcRrrSLv6++yIXRo3VQkp6UB9YyxV0rjOafh6+R1IOVBenB3GEbN80C9zfjDTvoIopMYtp8sagMvkEHeMRJ+VZ5+K0Z633eq29zipyU1oumZs/nZ1k0nKsP50ghO/Xg5d3N1t9bBKLfI0t57fz/A+SLFCSo9aI5U5sHNIOthEFjFE+9fmpWouzeyQ3IEDnIOB6nAL2K16sL677TQbtwBDBKiKvrF1bwLVRRw6WaHXrYhlqlcVF5KAZb2uyPuSF4Kjvl3ni5Zc9OdOglgZDUiNJ9WU4X4/rOodAAnnSXWzQwtWbDmT8yEWkfdpIu4ScGRf+6uzMtp1P+l7b0CBFPCskFfWvfhk0hCRLWEM0r6s4014Z3xcIXgfqD7Yu8cX8NDBbzlHfJKBwNzBD342EvYGE9xT9t4ukpXjOx27Lw9znjKI3uR2a7ztrd39usz6kTMEYNxG61xrLo1znFW9KkDHQXy7vsu8C4lGHiSZ1KM/ErZzYtshZXcEYKl0bWXbqo0EchU/1bvx6e5fYxtUhOodRvI4gNE66JR7QpncC7Dg+DwzVzG8u6z8k6TYKRUnROiGgIm3RSbXzPZZDxjx34j0CWl9cDLnPuf2mDrY5BDRn/StgeqQ9odNRu9iKybpjgtDAN6Feylqr3uL6vXxKPH+7ii6iLRIhPexe5hCbElSRjH4u7Bg9kGofHx/Ws/SRtQ35UOBS8HCJOb43WVC5f5WszIv7c91Rtjl3kRMeoTnBay5aKoSW8jMP0VykWNVmUrquaalHVXsabengnT60GusYnMp44EKgbA99du+Rp/EFvnQJrVMlovS0nlNDtfFd4z7LmgBzda+QcbaecDWJwC1q/lVfe6T3E/XAs67oWF4L/+ioAqR05o6MMFRKU56e1yLfpI756sRAintHN7kImS/IDy8ltU2JXZB8TN6+COBa6vSfcmGBELxC6ibqCS1SusdY3Gmk0skFNeQ9IcaTeecj9pZ3PCyLnkgxrpIowov9RvpmUwEV+PQFmlAd2wn3ubMgRj7Hwxq9+vn3/AN3/AN/6/h88NdvuEbvuEb/ovhC1Lf8A3f8FeHL0h9wzd8w18dviD1Dd/wDX91+ILUN3zDN/zV4QtS3/AN3/BXhy9IfcM3fMNfHb4g9Q3f8A1/dfiC1Dd8wzf81eH/AAoJbvegL055AAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Captions: ['a diagram', 'a dog', 'a cat']\n", + "Label probs: 0.00 0.00 1.00\n", + "This image is about a cat\n" + ] + } + ], + "source": [ + "image = preprocess(Image.open(\"./apps/pe/docs/assets/cat.png\")).unsqueeze(0).to(device)\n", + "captions = [\"a diagram\", \"a dog\", \"a cat\"]\n", + "text = tokenizer(captions).to(device)\n", + "with torch.no_grad():\n", + " image_features = model.encode_image(image)\n", + " text_features = model.encode_text(text)\n", + " image_features /= image_features.norm(dim=-1, keepdim=True)\n", + " text_features /= text_features.norm(dim=-1, keepdim=True)\n", + " text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1).cpu().numpy()[0]\n", + "\n", + "plt.imshow(Image.open(\"./apps/pe/docs/assets/cat.png\"))\n", + "plt.axis('off')\n", + "plt.show()\n", + "print(\"Captions:\", captions)\n", + "print(\"Label probs:\", ' '.join(['{:.2f}'.format(prob) for prob in text_probs])) # prints: [[0.00, 0.00, 1.00]]\n", + "print(f\"This image is about {captions[text_probs.argmax()]}\")" + ] + }, + { + "cell_type": "markdown", + "id": "3f19339c", + "metadata": {}, + "source": [ + "### Example 2: Video and Text Feature Extraction for Zero-shot Video Classification/Retrieval\n", + "\n", + "In this example, we extract the embedding of a dog video, and the embeddings of 3 senetence. We measure the cosine similarites between the video and sentences. We first create a simple video preprocess function to process video input" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "530e4d78", + "metadata": {}, + "outputs": [], + "source": [ + "def preprocess_video(video_path, num_frames=8, transform=None, return_first_frame_for_demo=True):\n", + " \"\"\"\n", + " Uniformly samples a specified number of frames from a video and preprocesses them.\n", + " Parameters:\n", + " - video_path: str, path to the video file.\n", + " - num_frames: int, number of frames to sample. Defaults to 8.\n", + " - transform: torchvision.transforms, a transform function to preprocess frames.\n", + " Returns:\n", + " - Video Tensor: a tensor of shape (num_frames, 3, H, W) where H and W are the height and width of the frames.\n", + " \"\"\"\n", + " # Load the video\n", + " vr = decord.VideoReader(video_path)\n", + " total_frames = len(vr)\n", + " # Uniformly sample frame indices\n", + " frame_indices = [int(i * (total_frames / num_frames)) for i in range(num_frames)]\n", + " frames = vr.get_batch(frame_indices).asnumpy()\n", + " # Preprocess frames\n", + " preprocessed_frames = [transform(Image.fromarray(frame)) for frame in frames]\n", + "\n", + " first_frame = None\n", + " if return_first_frame_for_demo:\n", + " first_frame = frames[0]\n", + " return torch.stack(preprocessed_frames, dim=0), first_frame" + ] + }, + { + "cell_type": "markdown", + "id": "53653e7d", + "metadata": {}, + "source": [ + "Then we use the encode_video function to encode video to get video embeddings. And compare the cosine similaries with the text emebddings." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "9fe4ab59", + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAGFCAYAAAASI+9IAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8ekN5oAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOz93ZbjOLImCn4GkJTcwyOzqrr6dPU68w7zOPMy8xpzMfNefT1r1vRF9961d1ZmRri7JAJ2LgwGGEFQIuVShEemWy7PkCgQBEDAfj4zGIiZGR/0QR/0QR/0QQDc927AB33QB33QB70f+hAKH/RBH/RBH5TpQyh80Ad90Ad9UKYPofBBH/RBH/RBmT6Ewgd90Ad90Adl+hAKH/RBH/RBH5TpQyh80Ad90Ad9UKYPofBBH/RBH/RBmbq1Bf+f/4//++waEU2/w+frzrn8uZRzk3uICAwGCAABrX10zAxTBMyMGOO8ganM/H6g/csyaRsX25MoYvu+v3rMlstF0AaRPR3n88RRx+X8/UTUHAP9rX2N4GjXemr6AyLFxispv6faUBeKRDgMTwARwB7Ao1xHjxPvEGPEGD1OUab1KYx5rkSkZ3Jds3xjMJjCpqmic3wNee8Xx7Oez/Ucr9dM6/MW0rasodZaY+aza6P+zT5L+6ZlzrVDf9O56ZybzdN6reZ/I8A8fz/advsXY5w8Q/+895NnWPLez59ZjcvSeCzVqfxwK1fZMg/+3/+f/9fFMquFwkVi5O4sTX4izszIlmFi1COxNOne0wbsxGPuQgzMOfele1aXnzPcW1K7HVOmX0/9NfOaQGZcOI9P5CiLycyPGBuS78a0ZS4q42ndd4552OvXCoFzdd6aLq1R+5tl8PdqzznFxra1pRQx8+K1lgBoCYSl57Teab5n42veoritodVCYcsDbIennwGA5y+K50xiUk8qc/G5tr2Lv5wjvasubycycj/0+za632Jc/Y6YcU+h0O7jkkBQhpjaQ+22kWr0k1tFyYiIiBwRo9UARRu1z7LzqdRjRPsdh6TFLM99XxIErbXVKnepLfeiSwIOmFsAl8qt+X3GU+xrbQiGLWPQurdl3dVlrx1nAlYzlta8eqvycDtLwZCVptMXEgGsgzmmg3tfFmaeuvAull7uNS99bU/ej0W0nba0nfO/M1FBKgx8Yv4OQI/IDGaHAJkbMQJjEgiBA0KqKCI2Ib7pfIqNp9+etgiEH5mWhF8NtalAsAKi1qhr2sLsiByoeqaFjNb24ZyytUaoX9P2LXSPebNaKNiX2hwMoiYvmJYlEGGC4Z2zEMwVxHih83bMrcKw8V1Yc1bvrZtj69yO/60rx9j2ws+ZyY3S9xOyC1bfvFhlfdkfW42rNWYkKyFGRCaEEAAAkZ2puzUhLzZtNb1lQV7yV9WwilWy9Pq1jOat956Ddus/C5fUloFi9637gesEQ/6dMZ8zV8CxrXFfquscjLNpvDcJvyWo/nrf03pLIVc8Ne+zQMjfl+QDZ4tuBtJcQDPqutZ08WodPv1PndqisaaWc10vXQEf3YfestBv2o6Zw/hCGTIsXLVG6GxRJ3BiHOQQ4cCIYDhEBiITYgSiLtaZSKjsDwLA33+cajqndbas7iVc+t7tWrpmf6s17ZrWwkd1+XO/zZ6TrczrhfcSVLe1vs1CQW5aVaylSCw549fSBqGQlhdTwn11uckSZlD+jW0jqCxyptRXMo7O6T+LxN8A+mXbFtKGqyQo/c585v3xlu9PtMJSIIPHmrkU4Mzc6OU6EZhkmkZyOHGX4CPCGHyyFBxCcvZEoikglRdNMM90c8FAsw/vjr6H4F/LAFuafk3XMKulMovRO7xtcS45mGOMzeiya8Z/S5QasE1naY1n/a9+Xtv2bZZCBmMNrpI1awKcYfZVo2Rg5Pdp4whElyceZSaNOZ4zK7yiP2vuZ1QQlx2D90XvwUrIdLEpRsInYZtmRpa/tqLM5JkR1IHMJAIhxmR3FM2U7Xyq23IjgX6N9vkWyEbvn/vp7v/uL2nJl6wCG55u/63rXfJHnGN2S7RmRC5ZW3W/nHNn3/uSoNJQ2pqaofVYB7/Wz6zrb1kNNxcK9eC1tZZlLYCyBtiQWARYbIZU22u8MMWS50+8Aakl04gsKv2/rmrbn5VN2fyw9dFHm6rdTtT82CzEZOFHhYvUqSwUU7x5hMPIYrJGJvkDgUHQ5TUREKkm+WKYKPNU0Ocfvh1zbX1ful6vuVtYDFstgHP3XmKUS4zpklC45FNoRR9N7qmatQTJLT2vVb5V7lwddf9t3U1BBN7k8WsJzaW2rH3nm6OPFiVr5dhpRRssbV5jdSrY9clzR0OW3FaTv9BeclvXekvbkctvcSxaA2tVK+7NuO9JNcx2ltSCBCJ5gRkBjOgEuWMAGLKgOI4+G4sMB5DLwkGfzRkqiqYR0egeduGV52+la7DltRj9JeZ4rWBYE32zVH5t27VddrPZUqDKUmjn2r4t+hTOtHNN3bUA1nGorZ6lz+eecVGoXgkfLT1n6zy9SUiqQDuUufUyDljdA5T48x8wBHPronyLULlEm3HLbzDcKkzPl8nmWYGPKlKlgZkRMhNJCgUnfxZpFAveJbx3ju45L95Kl5zKl8paLdlGGtXl7zUGW4XgUh1W+DBz3u1sqQVzLfGI9/zOb7ZPwcIdy2bf8iDNtKhJOc74PhEV7aJiOq2a22zmPLU0ukum7CW6RqvcSmvv4fy/C+W2wF353aN6EQttSgVZbyACqANAkA3uHSIYkRmnAHCMCEwYg3J9zjvhRRexGmcxJS8PiQWc3Gpo7ZpFfc4BW/9md0Dbz/aepXq3aM3nqG7XpftqJlhj6VstjUt0tm1cri8JhrVtqy2GGnpq+Xns9aX6bTvy72v54wK1xmSrxXD1juaZmVQ9b63mah2E84fqP9NnN6MCFqGkVp6dy7RWCCy99JZQ2cLol5xQtyACZht7Lt5zAa/Ua0wo8BHaDlFOUUZElEJMCUQOTIP8y0BEhxAiAke8jkAIjMCMU6sNDMCZ8c6OwTU9M4sobss8cw3zWlOXZWSWAdXz/hxTfKv/wkImW6ilQetzWm2q58dWp+hSjij9zY5lzRxbsM9WaK31ua57yVei/Z20h3li7bbatVboM/NsDNbQ1ZbCVozubF2X4toNw3/PZtcHVbSI5azAeBrMQgS/2n5rcOEN/iTenuDwXpbiOQauuHZLU126/1J99bW3rOet91/C47fAohPGGRkxTgXbW/r6Fqi45TexkUwzxr/BAXnp3W4VCMAGoWAXDJn/Z02LyChdjMgtTdcBSwvvEo+w5RpV8GLN9wOYHZD9KAtg+BtoBSD/HokhY8LztBUyLwlMDoADgRCpB8iBiRDRQzeqBe4QOKZUFQ5MMQ24b0eZLkz6GMpszQkbuR3fcQ8r4Vo6F9HSgpMuQTNrLIUWNNJq1636fs7yXrIa1vgjhAkSONKiQNB6675e6lur/5fgmZYVofsgrEVoCizO57q+5v3m+jVK9FWWAqNoYPaRC0hSfWfjGs4OQk3N17b4Lu+3eKU3C/X/gPz8ZhSBwobTginxvuJHIJfkhwecTMOAHpoYL6IDJ+VCNrCpktHQHBcYBDHl158Z/sJ72QYcfVuhsERbYIRW3Uv48yX8+hIT3eKzuLaflzTxWiishT9b9deC6lyft1qEM7+EfMnPuyRo6vfW6tNWwXATR3MZ+KtruEvRD/p+xGopXJgTeSGg2Eac/mfnlZZrZFlfbsPlx19NWxfaVkjlkk/pLTDqNYzie9ASDLZkLUy/E9A4TwG4DFkt3XOuzDU+iUu0xOi3+Khsu9b6Kd8sFEqjr1t+NexDK2qqy3/Q+yLO0sAsUhQLk+HB6ECOwLQDJ6sg8C7lMmIEdggslkIE0iY3wOm+AhbpEDnl1GIuQoCtQNAZ4k1bqGrZdroWN19LLSz9FgxHHdaWibbgKG3DWkZyS8vpEuxxThhoe4kKzFgzyDUWyjn47Ny1lqN923ubbs5tlljw5y4JOOuLWkObhcLioNxIQF6cWrbA+1d2/qS08JIIAFGCg5xEGsHLHwPMPu1UlnxG8i8lmClZChV8RNabrI5tMp8La2i0c+rYu6eCsdUpvVT+GsFQM446wuiSBrzmmUswzRZaG2GzVG5yDzPIzft9jnEufa9hpTUWw9XE63xbtn2W2S8JvC3t2iQUWgOaP79BMFDaiPSh9f85qJXrHhCtP8b0Bz0kp7oXaZolZ5yNxLHONSK6K5u/pcO1VfeSY3RLHeeu1xE+5xyx1zz/uxO1LYNWHqI1Dtmt2vZbqFg7y/4LLdMSCEv33Rw+qh9uSb4XrWx58szBIQIhIuYFfAlK2q4p3VX/u/DbW0yZt97//cgp7m/0BPEJaEoJB04pTwQ66hERJeIIcjhOCJQQoomrGqBY9r9bfSQ5qN3EkjAa3mROcvvzisSMuegdhcKl515b9px2fG20UcuJvbaNCmVZJrgWL29ZJpPPJqVO62xnW88lf0Bd5pxwWGtVrS23dEZ2Xca2r51iaP3cefPmtfxdLi7+XkqdN9tmdd6EvodguKeb8/2SirLcc52soPwfK4yUfmcmMMsmNmZJdscKG7GthyfPyZFFbCY+2hZCzNFPLdtDWrjldd1TKCytia3PO7dma4FQM6qtjOSaNi5ptUsRUJvqRjtN96U6LoWuLoV/bqGLPhMUgWeFXwsqa1lzrXHb0sY3ndE8uVap+G1crK39Lpn5zasE8KVT2GZ0L437jvAEtrd6y9w8g7BfbEvrO4zzNkP8pqQqDUSubEDLFqa2P52mFuQQHbUU5DercxhGbltQQ0YTfHbNiG47NGnLQruVENmihbc+X2IYlzDzW7eRiOC9b0QPLYdYrnrGhUMJWs7q+c4aqj7X19r1tb6338F8/pybUS3mbk+xtLmlrLWwNTvCm4TCtABQurTUtXbjPLv1+C9fXtrfimyLL1lSQJuJLDOWDWOCbYKSiNpSYXLNZLW0z0lavrzuykxN7J4gOYccEUJKVBf9gOgHeOcR3AMCdojMGHmPAIcRAQcGQpRzlSPGiuUnobfQzSKIjMY/6U+cWbOTbm/UpqQ9G8d8BSm+Xy/kJQZ5jSZvGYmlW0Q3LaW5WGqL/dxiem/xaeR7OSLymD4Lc58+RgVC0TwomaLFmillRaUp/SRqTSsrBKpfjIVbfivKk/PLbPmSr6g88/p3eTuhYGhre+ag0pm6sWxZfA9aOy5LZueyUxFYOypvjUjJ9cweSWd+V8FCk99FW099NatFvjtJSEQ++Rc4p72OjOxYjsyzlBPtd964tjBkWU68n6mzGcdfUjy2rc37n9621VJQqttWf9/ez/Q5/08+tAWPwjG2fVJ+KhSA9mw817YlyNL+9ra9I/XejbdYpjfLkno1bcRJrtYW3jG1HV3XAEj3pSIPaOqfpekkzxOSKEcZNSdpWq0lamg+ueW5C5bN1vZfwNi3Mp1btaXFHJfKtspvfe5aK/ba+m9xb+t9nKv7rG+yWl9L39c+Swzt2whW+xy1DjnG1drLuSi1a8OE7yIUtjo2flx6H2rnrZyBU15/CS6kNm5LJBmoqeQ4EvIguAQ56V+UTKixCAXg+4jCrXP2mvLnvl/67Vba/Zbnfs81fG4Mrm3XFt/KLWlJCNfXFXobN/tM59Tyz6ydQ3eCjzZoXXeed/c2ld9CrciBb/Xc5rPO4KJKKgcW8XmivGmIbRkDNWmPYxIGsjchipM5smhK7cdvou0a4P3GvzXm55635dD4re1upeBuMczvvXbOwWUtuGSNtaD31O9j6cyK9hi8PWhg6X4Le9267rX0/eGjK+hbmvkrnvCmsue7cr2/4hIt+RTKZWo/n+qPy4x3GhZo/k2RHioEYlQBUfwKVYWb+3Lpt7cunK1jvtZPcKnuW/Tnnsz+FuvtnPBU5r9FCNTte4uycI0fZyttqfoe7fghhcL7obcJhFvRN9PoqHg6aEFolOsVEyTZsMYAAgghOZhDEgqyP2H+SEfuaoNhCwPditMvbSza0pZz7TuHFde0ZZftGljmLfPpR4CNW/1b2tX9rUmd2t+zLauFwnsYMKX31Jb3QrcQDOd8Cplx0PSX+SahdJUkx1FxyHGOAtGY8BhjSm9cO5YBaN6aC069S8z2HOPdAufUdKuIr63PvYW/4V7J9m61Ls85hNfuBl6iazX9Wyte5+t7+0FHb6H1lsLmONP3i+W/N/oWTsQ1NIWPgJklQDSNQuX5jtTsa6DyWSYpFUcylzxHGqutZUo9qY6JYJnTWia55tot4KA19126vsbP9Na2r6XvqYAtPXvtfp9L/Wz5Gd7atjV01pd4hcXaqv8tbdwgFLZVvLZf72nPwY9M97YU5mVdfUO5LV2WzKYkgsHATdmHEFFCUVH2QdgwVqf9WgyBXb7W+u1WPoZ7ChCLm7/12VvoEk6/5v631qG0xPjfhwJ1P0HJkRN8tK782n7cJc3F5i39d2L2WyfctRrdPdrSordofLd8LnDZUhBIqPplxnCnB86fmwa1dWDrK++tsk7OtH/pYPtLQuEaSOEWzv2l72udoTQbq+tpqx+jRW/1SaxJFXHL/SG2zq11X8OH7L0ty7BcW59uxWZ9tdDa0jPW0LtxNP9Z/QRL+Km99q3ook+BqsnKbWbKSBOUSlI7hiQ5KRvV9K8wfiKbP6kIg+b+0ZVa9NYNSmvH/Bbv5ty7v/SsWtBdI6DWrLlbr8tLG/Pq3+65qe6SkD737K0C8NK6LgJ+G3qy9O639KWmq1Jn35w2vvd75zS/hjFc0nIu0S20orcwqjnTd8ttUOmRuPZkHJBdyogMhHQ9MCEyIYSIMbKcqhb1ORKV5ByKpYDp4rhGKz5X9haatc3zU28WWqP11r+twYLXQGHfi2rttO7Ppfl8D7/PW2jp/robLUuzng+XnpGhN8g2z9WmwsLz30L3PzHig66iP6vl9EchXeQ/ilP31tQSDveEcj/odvRu4KM/A70lLcK7oFmbFjSpyb/qNcv5U9N9LuWm1GtLj3wbTLKW3lr3JX9ATW91mr7H+XFOq7+HP+BWtLVt4jnbXud79m1aeh9CYSOO9megbwkdra130qLscJ4/V53RnL8UgSB/LmVJBZDyI+XUxQt13kLzXgNFvNVJeos6L0Fe30pQXkO2LTXMa09YA2671+NblhUV53L5rdClBGZ8/3f5PoTCB/2AtDB5aY0etVRluVcdz2/Vpr8VJPOtnvXehEBN5/Btu2t4rZ+hVffaslvLr22LlF1d9ezeFuVnMwPOf1cVeX1CvK37FLa25Iekot2W74kIU48UAeAtJyApzPLOiCb/TH+ifHpyKUzKCOjMn2peRVOSKAxNDtY+c/YStSCaWzqetzhBt55+1arXMrofQTDYz/Wu5GvH+56WwuryDMQNzb93YMytab1Q2Fjx2vI/tm/NxuFXHalj9C/E7P8IlJuv0Ue4zKA4lz8jEJRxoDCOIhCmsMy1zPCWUNGWZ1z6be31Fnz2FhjmntQSAC2fy733HG1RHrYwbma+qCQvQXwt5eB7hqG36McSYT84bZrQG2OVvwnd+TETxlck0Jvq+1Z06VlvEUrvHTKyZPeg3JLeV/+vt0B0s9lFGOk70jvxKbynF35fupdguDdpDHW5oP9UGi3UKkiFJj/LwTrTepPTmQDd1WCBMxmuuXa8htbCR7eAjuz11m9L8NFS+y7txn5fTFJIhYHVilvttOXeQz82t2GD76G2kvT6LZMH/uGij2j6vz8sbZ9473s8zguspsfB/DtnhJx3w3GeD1SVaX0+R2t3kV66tqXuc4xuzfO2WBzvET7augnzPQiFzbSiya15cI/3dY93/92FwgctUMbt/xwk/uTbY/y3TKK25nlby6+550djnDaX0JK18CMT4fufeXBPutsZzZscN3C4l6VAknHnLnXLpGjVvT6HzRIRQ7GTFQ3RPQHm/vy/ckUzjvLsN2Qcv+QemrebsM6AcWxjuT2YGQHAyEAEITrNnurgUoqLtNTMg3VLm3FIA/B+/bw6F/GzRiO/pMW3wiqVGdYMwyYu20r2vrXpE66p/xzUU39eS+fuucWYLLX5WloT0MAMuCtDaS1stpQKxZn0KWvo3BhfE7F2lzOat2CFmXncS5vg+2Lz+i5s85f3oGw7u3ptu3lhT/AM759w+lZNNvfRPNLFpDu6KGcza08SiJkQ08CwXnfK7M1zJjDT27TkW2ioS9DQPSNntlo31250PAfJnYuWWXr2t7bIWmN6TcbTS8+Z03UZcte913XzZK1/4txYLdEHfPQGWpqUNV07Qf+o5qmlcwrBnFHJOQyr5eqfYPyA++9+t4ylpd1aS+J7QkX3et9Lltpb6zkXVNCit/KWtWXvBh+tLov7Qud01yfM6z7nTNw2obJe/qekPxoO/aPSGsZlr4UQZr//KLSkVc/GYCNPWRqz5pi+AU67lVC8C3y0pfzddTm6L3y0hf4smus1tD4SKP7gGx7vQ1tSNGyhH0U4X5M2o0WXotwyjLpxWDaP40omvwap2Eof8FFF217eXJtf8id8CIQ2EWE2YMuL8WMMW3QPCMHSWl/KVvjoGoFz3gF8+/nRctRy5E0R4+d8HzXlDABY1v7POZWXft/ybj6EwpuoFQlTC4ocz4NtdtGPoaF90B+b7s3kP2hKaol8T/oQCg26dnLX0rgkggP+7D6CJZJQ2ktn1hp4YOOS+TMwqi148tbxqOd0/awl5+nattzTWrjFu18KAX5r3UswlXnC5nfaKq/7RLbMke8uFO7taP5WtBRzfHV9Usmme35IBphk5aqdsD9g974V3fPdL83l1vVvIRSuedZauhQVNFX0ttVradF3wW1+uMRf7hHmfLd9Cu+FtkQKvKd2/yEk5Rvoe4c3ftB2+tH9ZvUeh0UN/ErBsIpSYMw9UmF8V6Gwpfy3mEffQ4v6oHXE4MkkOAtHMG+Cj/4sQuVb9vPWFvEWTXfrPbeie6xxy6gtxANM0RO76e0tZ3Jsoe++TyHdcY9mvInuGdGxtvyWau9lSmvdGdNPM/ZcmG8pT2ZSr9OAWikjppVf1ZWrqe0nOp9a4Fxdlpbe2daY81sobEv9bJVv4exr2jzfjHj7l7ll7JYgy8uRVrRpcW4VfHzFuGwd+3N0M6Fw9QsmgBZwtFvQreKJ72kRNOuW2LS7PfMyFRM5m9XTy01SLYdIci1REgyOJc9RjHH1KX5tzHp9D7bm1jmHndcM4xJdux7q8Mdz7bsmjr1m/MyStM45l/OVWYF3rh8xRoQQ0HXr2chbnMtrGd+a8NlzpP1fxv1vDx8VZQOICM3fWmPwVr9Oi94kFG4m+em85mlp+4umzYLhj04ENLnrW8Ve3thjnpQnchIQb6NtEMJb0j8sRoes2CT1Vs19STNfgtYuMYaWIFDmbA99sfUx80yoWvhC6/DeT+AP3dFs62+R3ncri+ga4f+tYKhzysSMyedP7Xd7yZq5RZ++e/TRj0wf/oRraVuInL1Poah7YMyLmuEdqNZqrxEklzYxbW0LMB1X51xm3OM4ZoavVoUy/Tq1Rau+moldu4nurbS4aewOzPVedN3aWU/vQijcNUvqOwnjuc7Ceb8TcwsRJRuBW58J603x7XDJ5nYuMMjW92vqb32vhdESg7qGcZ0rU8NE1hJQxh9CwPF4xJcvX3A4HAAAu90Oj4+P6Ps+l1VSKElhKa3fWh0qZK5Jnb2l/Brmr7QpzfoGZOOttObd31pAvAuh8EF/TJqupQJRENzNw+7+CPStlICa0UwCCYwm/fXrV/zrX//Cy8sLQgh4fn5GCAHMjP/yX/4Lnp6e8PnzZzw9PWEcR5xOp5mlUTullyCwNbTljJZLEFwtBO6tfW+jt/vB3kIfQuGttPqd2IiFOnphWzTDe6eyIOcwkVoKqmndOsjgXBTN2vuvvXcLnfMjnIsCeqtjsaV5ar3KKPXzb7/9ht9//x0vLy8gIhwOB4zjiBgjvn79CucchmHA58+f0fc9vPc4Ho+LEJFl6teO7bV9PSeMvoWPYXvd83XzrYTWh1B4C/EG6IMZOVdSHb1wh2iG90Q1ZKTZxOzn9ZVdLnItXr0mDvxWjGOt1twSqlr+2ue26rYO6BgjTqcTfvnlF/z+++84Ho/oug7H4zFHHKlQ2O12AIBhGLLzWP9aTm3bhntbRq0xWnLa35O2wGTMjJCac+t3v5Y+hMIHfSMqVoNzEqJ67804WxaP1ZSBZbz/noxsK5xyLY3jCO/9BEpRJk5EGMcRLy8v+OWXX3A6nTCOI3799dccZbTb7XA4HPL9fd/jL3/5C37++Wf81//6X/H6+orX11c8Pz9PrAMbmbQFCrol3dpPdG+6ZCHcw4J4H0Jho0n4Xl4kg8HxfbTlvRNZ5xwXzTEybUDOlsOLW3Nia8jjtL00+1PBca1waGnpSyGGLThpS8TV2n4XQS1MOoSAw+GAr1+/YhzH/Hc6nQBMI5LUavjy5UvWhvf7Pbquy45ohZzGcVzd9nN9euvaX4K2tjilvzUtWQz3ou8uFHL3NphX2wblji/2jlVPmOgPTNa/UC6iQEmb+nnZX1D7BLZYI/XCsxu69NqtqBYIaxb9pTDUa6AGaxEREU6nE15fX/Hly5cchhpCmEQL2TpDCNnnAAA///wzdrsdhmHAw8MDnp+fcTgcsrO6ZYHdgy5ZBC2hsGUMt7Z/236MuYvx0ru/JX13ofBBH3Qv2qJZ1uGZQLE0tuzYvYa+FWykpP1xzmUYSRn+b7/9hl9//RX//Oc/M3QUQkCMEd57dF2HruvQ930WItoHZsa///u/46effsKnT5/w17/+NUcm/f777xmOUsvkFpr/OVqq+1tbAJtgTABb/Iv3mDd3S4i3iXh9CMrmVvC1h1asiAFfqJkx105lXpjoI1zaxMOl/IID8nzbWhdSdAkttfx8TPv8YumHqjdJtwIAeBoBfgUA9BQAClKGGCeStkTyCPmwInXcm76n9oo9QXDusoNQmU1tJVgm1GJILf+BWgoWRqrvvwWcVAukTfWRjBY50z79KbWT82uS9xRjRAwRnhx85+GdhyOHr799wenlFS4wPu8f8fz6FYd4gqcobyZVNAZGZIZzEYQAwglEr3h5/oLeOzhE7HqH3W4P7zv8/PknOBCeX17x5csLOO13c2TYIKVxTZAsOSozYeld691UX2eAl6xEM2/zDQ5qs2oalrJaGxaprnHmyaoua7WkeNF5WM8TZjsHLDTKxlqon11/n85L0oHUX69g2989S+q995bxps1Ry9SqgedzMb3kVqdshNFWCTjXJC1z3EqSa7QlaKYPXzRZ2X5QPwEnwVDOo4sc4d2Yf+bU5hEjPBVmW8ibys2CJnu9lD/nrNR8PjbMMrc6M8qpYPDel0deYPRrtL9rokWWMO8LT0rylPJLrGca5dckCliunxmOSN5ZjDi8vCCeAjpy2HU9xs4jBIe+84Av8FGMMs+ZgREBRA7dKSCcTjgdDzh2Ds9fPcARu90DHh/3+Pz0BILH8cBiYXDMjJNSeBqDEdO7J6a8xs6NBqtUnF7E0rqfjpCSCIWs4yzeLQJBhYLN2muFF8HlHGByeckanAqJ6aVLAqG8cq36FoERH/DRG+jSZP2grcRGO0rMERCNT60bjqu0afUHKKPXyJd6cdaY+lTrmp9NXGPO97CgN8Mbpglro2ucc3Cp7c45jGPA68tLjkx6eHgAILuXfe/x+PiIY0ASBgZiCwGHIPc47/Mu59NpzOGs+/0RRB32+z2enjzIDfjnP/+J8RQwhoBhGJB1fpJ8SswsG+UIoCpZ37xfVuGy174faftsxNX899Y1tAyfVc+71Vy8m1DYqun8kC7VZrpPnvSmhgXKZNjieJozsfdB1ipKWjvVh5pr/4sZ7lHORXBg+KSxBwAxabE2qItcivgBAOfzmLYigexnFQg1IznHKMkwykkvKqvClm29k7e8p60Rdqptr7UytKxORSJJZPf6+oqu6zAMAxwRjsejwEqe4JzHPpK8n8jwvkvvLMJzxINj7Cmiiyd00cGNEeHEOBxfEQ8HdHDo//pXdH7A58dH8F9+xvPzM14Pr7kXQAqP5ShfvQNzAHOQpdYYUgJAcb6eGAxurk8rQe3lUH7O8BE1LWpQ2a29qBikqDq7/mvBBpR5VOePuhW1otku0Yel0KC1AyiTovV9GTaQumWG19J9Up4IS814F4LhXBOo9bEsLwKg6JeDAEXqVir+gFJHQcraeYHqxWn/takWVIicY5rn3kk97ueihNaUO0ebhAIXGOOSYFABUpdRH0zXdYh9ACf4jcgD5JL27pHBlzSOITh0CNj1HYbOwYFBHIEIhHFEOAExRDx3ezw+PGDYEbq+x6fHR1EUnAgfSZ8hlkicuFutxbjQfyyolS1ebX6g+peyNCelL+ntSzBhPTfPlbsGaqyppYBehq/mtFoobG3s9oVwL3Nvi1a+DfMvDKZow0oKsdfjEOOcyShkMtF4m/jhXCvZqlVuo631cvVvNQeMeQwzgcUKiBD4iBFCzJaCOOoAlyDf2jqImXnNIZ9WCKU0Y85Aa8ESQsiROVsW1Ld9P1NqOdFnbU9KS4wxvZ1keSXtd7fbibkWk6OePOCSVecGOOcnSe8AoEdE33cYhj4LF2bG8XBACAGuO4HRp/0LAY9PHT59+oRht8PD4yP+8z//Ey8JukJqi9ZDbrqu5MHNzk+/u2r25Uqst+ASZl/KceNqi4qvZsqk5/DRtFl5NzjHRjvO07mAiXPlluidWArvBZlfsFHPlj/3vb6uk8NqBS0GOmWu38cyiJi26VIbDMNvCTQwXDo8pONTdkYH9qDkXBzZQyOakqtRUAQAOaXIlUNhBUfXdTmxm/5bL2B1UuvnczDV9yaJtEmfK4Fg/82UlBWCCNVxHIHIGHyHjhyC82DfoX/swPEk8A0RKFkMu12P3dAjG73jMfm5I9zpFRyOAAg0BvDphHA44uspwo1HnD69IBxP2Dmg63p0uz3cX/6G37vf8PzyjK9fv4I5AAR4ciBXuPtkx3npPHKMEpfrFAH2amVYQWDHwY6LhW9s1JKJjEvBEWq16PzQf3O1RCDnQCgb/Zi5giTnvIaI4Mkh8Hoo6Za+BKW7WApXWQn3WmQNh87FtmwoX1sH56/PISHmuqwCvdoWre/tkMR9STuyBFtM4aP8mQAHAhyBooyPIwdKAyPMpkSo1NbXGlO5FZFhmf6iVm3K19/XvI9v9o7SWF0KuVWa5iVSZyjDey8O4xTa6r1PfNNnhuacQ+c9Op8c1cwAvHm3IxAjIjOIAWJG5IDT4YBD18H5Hl2/x8vzV/huAPkeBMmdFDlKQr1R/QoMZlFOdMzVqin8nLNWPlkyVn8gsmpZmaF2vtr6poNryqXPpkgrqs05pwbZIlnl0Ias3oKsgnNXn8I9TeD3xOC2tKUFF9nvS3UW07FM9PrlETEctUMubwNRTA1iyg0jvTSBWWcSr7Kn84KsV+akoLGAtL+p/S4za0DERBkPlwWC/l76vYTHnosWsoKh9jcslV36a9G3hpD0/dQCoSUYVLiGENKkFSbmgCwUiIoA6LsB3hkhCogWbPooexwgzDdGhHGUPQwkwiQGSbB3PB7gugHd8Irffv0V1PVwrsfD4yO6zuPRP0oW1ueA03gCR4ERyclRrjpjLLQ6mV3ZapBy2bELAlP5XPwPMc9lXqGd55oJpo72eJNzAOZ7ZabzlSb33SIf1DVCoKZ35FP4UamCTTh/WS6LeZkl5+Y9iGZt0YgpwxQNg5+mt6Z5JIgRiDQRjmUbnwdnQ9zxmJ2JTB4BERERPXUgOIkaMbuI/cQ6aG8Eyk+sLIb6NwATOEgFg00Ip9froyrPCYJLdFfBwDXqXf9etGkGwCRMmhjwXY84BvHhqMaabotRHMaeCIPvQJ3L79YhZUMNAeQiJJ8V0FEEOdkYBzAGF+XdMzAgwI0vCM+E319/RyQCE6EfBuz3Dxh2e3x+2MNjj9eDw9fnr3DO5/ml/JUBqIiw3SZjHdjPU1FhRYplwstWrhnmPJyE6SFCmlQwhJDCfUW4xhhzPigiQtd1eW6NY8wpRKxf63vTO/EpvC9a+2J4wiqNos3nmbz+ZC0GW1bw3vYkvdWkabDTWQRH7lOj/Ox+KxgoX5q49MiWNfUQyWY8F+X4neicQEqpImfrNdbTOYfaJY3J+gpUS7PWwpJAuDT+536/14KPZ+aJCgJrSWQBiIKHxxAxHo+ShgJshCUjNiwflzAcTtZBgWTEApHNlQROPgGROQFhPOFIB/R9jzFGjMwYxxPGMaA/HhFjAHyH3W6Ac4TXUfYyhHFEsCnXUSL/dK44EgEmhfL/ZO6R+VzN3rMCtUHMjNEw8/weDCwHN1Uu9K/rOmORhXxfCaDAJdm0uo3AdB6snX9/KKHQ1sTWa2hXaetmnIXB5BpygSXNNUOyzTbMZ8c53PqtWihlSMhCP+doXk4x6uXicygpsQ4ASTAwAXEqoAqkNm3WltDSui79bOuoE7bVn9+DFtek6lXkd0nlsw1ZtcJPo7dCGHE8HuXMhFAcqByjaPQ5KCLZlN7BwYE5gkck7J/BCPAZXtLIMrlTteKII7wjhBgxhoAYRoxjSM8OePz8E/rdgGH4BH49AMdjYsDaX0YkTn6FAi1m5m8EgI6HXZa1MjdZadWilN3S0m8VrjFGhNNptjHNWqLeObFygOSncTl3lAoFZsqO/um6N+2u4M+163wpPHYN3SXNxXtaPNlxu7LsOe1zRtTc2jJ/nLEI2lZDi3m5Te7xrXhkS7tc/8BlwbGsdTE0wkOSVsiC6fgIlz4HGuDhEEkclTpUHJOIIILrSzjkOdy8NR6XGHt9T8s38B6psBNlhmK2OUiuKCbGaMbIQ5zIYMZ4kpPU1Lr95Zdf4EicycLECIMHEI7CdJ0DvE8CVAw36jvxI8QAR4Qx7VNwCHBOHNe7nhAcEDmA8YoYRnhI7qMYIsaRML4Sjl8Jv//+K3b7R/z00094/PRXPO2egM/AL19+x+EgguuQsrcyRxCP2QG+2/fY7/fwroN3aa6kMQkhYGTJrZTXesKZOuqlFDMYo9yV8qbFKFbOaTwlrT7geHjNp80pJKTzbrfbgUhySal1oL8rjAQAIZQstJOQaJ7PVRXms3e/AKHqv9coi+8jId5Gujf+vrpOXmF48hrG0rq2vV9bYK+Le8gb0NbKyptEBkCaimnOmllCvFN5U5WOM11+N/ViWj0mKyyLLfV9a5r0uWFlWYHnO7mmGVABQNw4sps5jCHv0+AknFWhUajD+YbwZaDrfWoDkn9pmoKcmDEappz/EgNmZpyOR4xB8iM9HoH9/gH7/R4/ff4Jp4d0vkMM2adxPHxNp8LJWRDMjKEfMPSK4VNOleEZCJEnUE+MESGOabwsU40IQaCiEAKOp5JG/PD6jM+fP+dd80RkordcEgjmu/lD6qf3HYZB7rOh0YHL2RNa9ty8X8MPt/DJHxo+urdD9s3PMZDJDD3J1AYRyQLvax+3pZ2r6l4DJdX419oGsPlXHZxRGAkiwM74NCgPU6uPtRB4r4z7XlT3dx7JVpi3I4eh61JCOuQDcAgED0Lf9SmCKEFHaX+I+lwiMzhK1I5zSZimZ5AzezoAgQGpvBPnHDhGUAo1JZKwVk3PqL0IYUQcI47HAwL3CKPAek8PD7LHJEbEpNWHGPD6TCB6xmk8IgaJfIqpXc4RvO/gOrEk0r48xCC+ipj8GjHBRMknnxnxGCJOpxHjeMLhoJr9mM+q1r86BbnCRa10KCXaiEAkcFI+uyKGSVTcLKrpDFy8FFixlX5oofBjUP1iWthS6+XdT+CtRCXN5w3Q4apSZdemwxEaBdLxczqmk3HimBYwI6IHEi7MBjPecvbtn50ypu099sMuwxZfv37Fr7/+Cgdg8D0eHx/ROQ9GwOH1FTvXIXpJcid+hBQF5BQQZHQE+L5Dhw7Mp6LmcEhWRoRzXQ4n5hjByW9EBFAMKX9RsiYigAicTiecTozX5694ff4KOI8uQTb90GdGvN/v8OnzTxjHE07jEYBAlAOVk+BKHqySk+gURoxBfBnjGHEaxa8yRrWIGMfDCS8vLzgcDjgcDtI+R+j7bnKaXLGqOFsNKkhtfiO9RwRlsTKkbQJt1afb3SLMdAv9kEJhSQLewul6LqRxfm3rc9a/VHVqbap99aThSdXN+6j+ersJOamJi1BUDa0mBrJ2ytV+gtZu42ux1FtTy9fxLZ7ZshQyvJEOyZHT1X7H77//ji9fvgCR0aUQyvF0wul4lAR5eETvHvDw8JB26eozAkDVrCAJEojCdRHMEZ4WNtE1xjFi5ATj2A1wxrqIUWCkl5dnfH3+ioeHRzw8PEy0cAc9LEh8HCEE8Dji9XDI+ZwKtNNlWIe8Rw+W1B7kcDqNeH5+xi+//ppPi/v69WvOzaQWgWOHkYRh73Y79H0/Oc5UQ1OzT4w5X5/4DhDzb2p5MEeQp9VzeK118IPCR7dbxHbtGfBmw82zLOmz70tf23XacmvbckemZrGsSbSFhWGm3wW7wazcjGZoGC/8UD7n6COMkAh0htOpyYwAn6GjSCX3kWUw14TevZWWlIdvLZDs8ywzsdctpk2kx26K9htDAIcIBsmZykkoHA4HnE5DykXFKfQzOZu5jjST+aGQHyvMBHF0z9pEZMqpK5h1+QEs+1r0sJwQRozHI3jYQdOnK0xFgMBQRHDkEEmUnhhDYugjfLKUvE9CoetSCg3phO96ACRMfujhDsliPZ4wnorvBcyIjhC5+BCsVq9afqCpUKjfTaoKIcRstakjuaP5uR6XIu22XL9E70QorG/8Zu1rM3+IM943eaYVOEznm04AjCMV2MCweKrNn6PtTFAWmjTRmO2mGqoFgvnlUtW5PhSGL6mMnS0EAHAc85UeQEx3DOJhQARj5C5HjERQ3lFbh+vZBbQlBO+tdO5Z30JI2WfbCBUbemodvpom+/X1BePhiN53YBJI5+XlBWEcRTCcToKpR+AUgL4Tn4TzDhpoKow/ZJ2hI4fIAr2kIKgEE5UUFfB6AE3SruNY0mUTgWKQ0GRAHNoOcBQRxyM4HOGwB3EsqS6yowACyeic4ojjUQ7zceCJUAQA73uQE4tj9/QXON9ht9vh8+efESNwPI7w9AUBQdoXA8ajfI6e8ZB8HGOCnfSIUoGeRiA5jK1Vkx3bIeBkHNdWwXnw+4lvwjrE6/d9bi5cKrtE70Qo/KDEjLNMkoH2mQu3bsb72Al5DamhklhXiZYhL7izcTCL4y8AIUwEgOK5ijG/N/oe1kvruh6sE0ajlaad4zEEjFHi5YdhwG4YJrj46XSC70uIJQEpamcKiXBieDFp1hoqWtJkTBJEAEhwT0pr6r2XSKEkFEZmxKRJn9J+hTCOOB4OOdQ0+5iI4NJcGINAV6fTCePpiJjCSW00lu+GFAE04BAcfN+j73vshgF/+9vf8NPnn/Bf/vJXfPnyBV+/fMEv//kf+Pr1K04J6un7HsfjES8vL9liCEEioDQkteu65FQuUVoqCF5fj9lK0D0yzjvsH3ezd9ey/u5FP7xQWHLClO/bBvGST2F6/dJCT+rTxLooVsOlW7fi+NdMGHEEaiMvCDkC6hDbZpkLP1S6jAgGAB5RHIHEcIhgApjEdqBkcUWeO96YOcd/3yJ/zK3oe/g1FoUCBFpx5BARcnx930u66wA590CZuHdOGDADx9OI4xjQjTE5jVlAvpTf3JH8iYCPycqdAq7yNVmlpOxcDu2RXepO/iWX72eWUFFOy+h0OmA8HRDGI+BdSS0BA491umdCEupFTpvjTsfk5E1tdg6+C4lpB0Q/oCeg6yQ9Rdc5eOfhf/4Zu90Onx4fxdog4Pn1BV+PL5m5658KT5mPAzrfTSxXq+mz8aXpe5NNb36W8+zSPLo1fPnDC4UW3VMrqyf7RYyHpmWogKYrnrWtbVti8s038+/5zVtFZmwZ3wUmTfqMZDYD8DzmKCOPkMpopBIDSKGSpg+1Y9UK9XuHp94isOGWtCwQ5C14kuSCzjnRiHc7hJ3szCUg72Tuug4+WQRjiHg5juiPI7p+zDV2JGcfEDglzEuOYU7QJBcYS6yHpKVTAhWZU+gowfleglJT0jtOwn8MI0JEPlvjdHzGeNphPO3ALoXThoDjGJNQ8Oj6Ie3EFtiIWaCccBJfCUfJCOucQ+hHhNAjxhOo26cIJZnklGDKT596fP78GTEGPAweQ+/xy6//wuGXU96NfDqdcmSSzseH/SOw2+dw05yiPL0nsWp9jshSq6zrPMiVedtKqKfPsf/ekv6QQuFbEasKc5ZMDvgNDIqLerWK3iNs8hbK4YrOpZTaIixC0g5bQqFOYgf88calRWs22NkIpK7r8PnzZxEWEXh5eRFhQU7glnHEmKJinAPGkfH8/AyHgLjfJealkV/A0MsGMUeE0ZxBUTTjkk4DOaoo5vbIfgPO73aMctDSOI4Y4UBpZ/BuJ+GlIMrM+HQa8Xw44vn5GWNK7BdjhHeMXc+yqe14QDwdE0TDcEnwOTiEyLJPYziAfIe+H9ANyP0h0nBeh//j//g/8Je//AW//v4buv/f/xe//PILjscjfv/9d3z9+jWH/T49PSUBcMAx5ZNSvwMRoe97PDw84PHxKUdGWZ+PjdRq7WK+N72bHc3r+d9SLqGSd2ha15qKp33L9V6AStaNCRmteJrobrYfeuLP5s3+iNrxutiirLHp93mTq+aY62yK2Doo3zOdxq1JTXlsZcFPrzGAHZ8QwQjEAEUEF3Bi4AU9Rg6SqC2Kk5JDAAcGMcF3HXynZwIYeDGDcZogDvm6zqnWmb7nMPoW1cKqjo66Bzly05fF1iJFmoLJeZN8L48PDxJxdJANXwnAATjmaCTvksbMhOPriFeChK66Dt3eAV5DQV2eU13fA+TgvBzpmZlaCUsSZuuQYaEx+ThCjLKbmpH2MUQMfQ/vu4T1e0mLMR4RfS8J9uKI55eveHl9wfF0wikk5zJkk9zhcEA4viKejhiGAY9Dj13fpay8AWCC6wjH0wHkHbrewY979H0PuA5gyOY5pPW7G/BAP+G//5//N/iux6+//Yb//M//xOvrQXaE98DheEKIgHdHxCgJ/0KQ9BjOiSNe9joMIBrgvcvvS3ZRJwgOyKHYOqfEUDfJv1k2GTJzDsBwSWiKM/uE4+kkfpmVAubdCIUtRI0olvb3BsNbrnUqZDIGWpeqGOGGcZnDMTW6bqWCxN7ciibPTovyerK+hyzx5BkVVtpsi/nsqLXPAHDxJFEeiBjcCyIiDkz4D/oJQEDgKDlpAoMpgIPEq4PlkHmRCEYgm5Ba4U9JWHCJRmFHrVe+mWrT/pYmfmsdeujGLKMUVB+yPGcZ8/1uL4fhHF5xOh3hCRgdAA4YEREi0DkCswdHYAwBh7TjfOhPGDR9BOlfYkidRPRwjIgpt48VkOIrk8gCdkBg2fMQWQXDmFNkixXixfG926HvPQgRYTwhgtLO4hOeX77i9fVVfB8xitUQJEXF6+srTocXhPGAp6cn/JenTxLRQxFEAQTJ5Ho6HUCe0J88uvEFcBHkGTGkxIBguF6OFO0fHvDf/vHfBfJyHr/9/gVfn1+yf4QOR7jjqTjjTeI8tXTc6YQxjPBBICN9UcwRcawgI5hzFzDlOxzL/gcHEQwMwukkOaKen5/x/PycrZU19AEffdC7JJn3hek45+DhMKSwP3IRgTWKxjqxL2/6sSZ5iMIYnHPp3M/3o/y8jYylgAJFKIyz3+9FY08M63g84nSQlNaH11ccXxP0cYxglpPRAGEsz8/P+PT4JL6JrhPnMlJKiXRKGzODXQmpzMIBjHzYqiN0JEejjpFBpFYVUn0lVYTUUzaHMRO+fP2K3798wb//+78nx7KDT5vJyIlT/Oeff8bhpcPLs/ggDocDemLs93sQS0it9172RJh2ahTReJDvIEKXxk8d9P/4xz/w9PSEn376Cf/jf/wP/Prrr/j1118xDINkSU0w2dPTE4ZhyLDRMAzY7/d42KsfA2VjnPM5oktJ26Rjou/SMnlNuheTUPy3f/s3fP36Fc/Pz9laXRuE8S4shS2a1NvaceleWvh8jq7XArUrxVFq6rxCu1zCkmfXNtd8P1rcdyJ+x7SBSSAMT4R9H9FRxEiitY6jpMNgBIBHIHqJVwdJJFNMDugUwVRv7qKUszVwhDebht5CtcN70q9vRbMYAQnnpcQYvBcNHA8PGMcRnfc4mqR1SDuJnRc8/eHxASE6xBjwejrg92cnWj4RHoYuwXMOXTJJCAwYhscp5QUnr7calMyA84QuiulA3iE6efkSjSPnJMgNogDECHz98jt+//IFX5+/ivOYNXw54vD6gsPxiMPhJM708ZhhGece0fcDmIHj8YSRZI/A06ennBDQO5+VET3PO3I6mIjKXgMA2O12+Nvf/oZ//OMfAJB2in9Bn8b38+fP+PTpE3a7Hbquw27o0fU9+s5D9kQlGNV5hDgihASvmTeHvLmvnNlgcy6pYPn69SteXl7wH//xH/g17co+Ho/Y7/cihB4eVk2du1kK7wlumtIZrD3//zYwwqz+Rp2sk9l8v67uZcfq/Nr7iJZRWooeAgCkMxYYDE8Ou04UegcgMoEYCIyUGjpFKjFS1k0DcdHUgrBhgRMw7Mp5O4W/1mVcXaI1GzQn1868zhwOyQzEAB9F03TOgToP2u1KTh8gZx4NYzpBzIlDeBh6nELEaWQcXo/4+vwi4+Y9+s6BogMRA95pA8Wpm8Y4RLX8GHAKh0BSbDuCT5YawSNC4D/nHLymkE/YeYTkw3p+fsbh9QWn41E08uSv4Bgwno44vr7i9XCU1OFRQlFjDKCUII9IdnWHNA+egOyEJwOLuS5ZPRElMiqqZSQRWp8+fcLf//53vL6+4utXgbIo/fb582c8Pj7mNNt95+G9k3blnRac2yfvxhtnc3HHgKZCQecBEWWH9++//47/+I//wPPzc7Yw1Dp5enpaNf/uIhTer0D4fvS9xqT5XOuE/M50ViBA/Ecp9T/ADoOTPDPOM9jJwhwDgbif7FYVx9vc51Pj24wp/HRLWurTtyN1XhZYZEwRRsMwyL6Fvs9QkoILlGIcXl9fwXwCs0Av5EtU0a+//orXxJQH/zfEYUDsOvRDn52mCy56YyHLB2GuBOc4O3jL3T5HnYXDAWNynv72+zN0o50fdoKbpw1u6gNglrxGHD3CKELAUQnH/TKOOB5NojsTGaX5kTw62dsRI6jvJpCmhWX+8Y9/YLfb4eeff8b//t//G33XYTcMAl8dDtmvYLV83djGzPmgnZxi3LSFKO8fn6T83u/3iDHicDjgf/2v/4V//vOf2VpQmLDvJaT2b3/7G/7617+umjV3g4+u1YpuTxfaseBQXl37FeNSRwm1oobk0hZHs56fq3VR9d3UvanF96DijFbGYHebKsnYQBR9kqxIj3xAQEQgoCePE0WcgsOrHsTjOGuWxMkBigRokORVYmKBMfK4q7Z2Hdl3uMZiWFPflutNYsBGrykzCkFCOTUhnPcefTr6stOTwdKYCTP7guPxgK9ffwdTQOCAEE94PR5xDAHHGLHbC1b+sN+jI5fDOXsn73c+lw12lI5gJU/wDkmQd9A1OUZtu1gvx+MRx8MBLgb0wwDnHF7HAIQT4vGIUxjREeNh16PvB/yf//0f6DsPjgH/83/+T+z2e5D36IYBfjfAgVOkTsQ4SuoJ6iOinjPtRMDAWA+KJuTd4FE2q/3808/YDQM+Pz3J8aFJgIUwIoYA33kMXY+u8+h88sUwwBzgnTibHQiOOiCq/0UVmBS5JRMBIAcOQfwYv/2Gf/u3f8Nvv/2G19fXfBZG3/d4enrCX/76V3z6/IRhP90pvUSrhcK3w/1vTWvb8m3afGn34eodz/OaxdTMY19yJ81CeCn/7x2QcH1loOcsB2JGRxGOYjK/Ke1hcIgxpS52rjB8tr008BrFZJffbgwmFsod6j17Pb3W+Rqdzy11UIYgp6R5rzH7iTkTpcihgN1uh9fXV0lJfTogQCLCxDEcMabzFb4+Pye4iLB3Pu+I9q02pd3MRMUVbh2glBhejsaB7GEggpyVkHYPI0XbuORHEHhJznLOYbJpM9huGOAd4afPn7FLgkT8TPIs3/UJry8ZS3VudL7Mm3KOsrZvevAQEWHoezzs9/mwn9PplDL7Im8czB6DnKNJRKZLkUPZIqj8CPXcOrwe8Pvvv+OXX37Bly9f0rsqqUoeHx8Fvvr0iN1ul30Pl+gu8NF72eEJ4K5qsY2UuNgMXbiNBr1FSy31z5PAtfdziBP2vjJhqT9LD5XyZdG1hYQyEuck9NTDAX0PxA4xysIO6BAXdlELI9BxebsTuDW+S1bDtXUvXbNhvJoscNqWOnghRdfEiOPxmPMXdU42T3XOAymqxpH4al5eXtKJZimdRDq6U6EoFTC//vprjgrapUilvu/k/aS2TJUVbZvg6/P+qmCQ9BlIY5qProwSERU1nXqaK2QPp0n9/de//oXHhz0e93v8/e9/B0KAU8gmtfnh4QHO+ZymIrgD3OkE5xweH56Sn8HhdDjkCCHvu8lO5cPhgDCeEMYTnp+f8xkMr6+v6LwIp77vgdyPEd6XPivcIxv8XBbc9thcIsniqnDTL7/8gn/+85/4j//8z3QmhEQjDSl3088//4y///3veEonxK1Nm/MnCEm9H/ereMuFsiVFsCU2/0fjl7fQ98OzW20/LxDyN55n97RlexwAMAIIDIEKOt+hSznoR6QTdhmIRIhIWqHVRp0HQUIcHZUY8vdl4S7TuXYW4QDZFNjwqSgTA0d44gSDpCMruw5d36PbDej3Ozy8dqDYoXc7DCfgOHY4Hg6gKLuexxjw8nxAjMDpMIIPAZ8eP+HhYQc86G5dQA+xF9IMvaIXa9prZeTq1SUnsWEESB6scEKMR8R4AjMhjAQgYvBA7B0ce3Do8PD4ALgOgR3iOILHEWCJUDqdDghjQD+eMMYIdh7oerh0JOavv/4Kdl+yEvf46TOenp7k3OeuA4eQM8CeQpBUHTHidDzieHjF4fCK3377DS/PLzgeDggx4GG3A5JQRZRT7px32A1dOqOCgBjFynUOgSVDbWTJ/wVKeatch+AF/nt+eRHI6NffslXy6dMnPD4+4r/9t/+G//pf/xv2Dw/Y7QadNavn17uAj+6/GO/Xni2WQm0RXIKSto7KOSthgnVfUfdqWuT9y0+thdcSlCTVi2B1kEPfdUyZrDWWNhxlP7Oc8sWkupJK8xL5Vf69zgfQGvtb0Bb/gg23BZK45XlZ1fzHMWD0Y0kPkuAj0lPaug5d36E/dZJFlb3AR77sGKf03DAGHOiIr1+f5cwCTxh6AGR8BSg+pGz5afvM+9PW5/aQ7oJO7485YfCSQdV1Hl3nwWCcYsR+t5N8Sq7H6XTK4aCSckKyrAJyloG+eLVCDocDToHzXgGNmlKntUBPDpR2YevfMVkFr68veHl5kU1zycrqvYOjlAFW34y+qwTBxTHI3gbnZKd+CMn/lfZ6pPELY8DhIJlZn5+fcTwdc9t++uknfP78GX/961/x+OmTBBOY1Ntr6Y8PH11BzfY31iYt/fDu6T5ioWxOWrIUMtee3DNv2xxKApJ/zdTuSPZ8ExF8ghsYDiES4HQ/eBIwBCPAKdeXwLRlqG0lg1+6/1pago/Otoemz+eGQABkhMWnAIwjZShJz7vQKBjdpBX7HuM4wgcP7yO6zifNVfcSuIyffx1j2khF2A+JqXceUKgkT4PazJ6mJcn9Nf22qdG7vs9nGlNMfXAeAbIxr+t38MMe4xgEyupcZqLhJGmqQ9q3oZp+ZNkB/Xw4ZfhmDEVBCSHkTWLiQC9nIxSB8CzptQ/HfE7C0Hn4hPO7tPNGSYXK6XDI45n3VoojIj9fxviYQ19fXl7krG0AP33+CX//+9/x008/4eeffpaDgtwUmls7M//48BHRDZCYOewjV90m1ro5MuWC9lnX9z2pYLm8IG/sGC7hbvZ3oZmfIf3qwSCMwgjBGXvtMKDzAgm9xogjy8aqQDtE3Zymfj4GiBOXKmr12X6eG/OaaX+r9xNVUBLB9WZJU8yjri3RNoUYwWMEh8S4+h5DP+SIGcWnd4OH5w5EHRxOGIhwZIf9T085ad3pJNouc8Tr4YB//etfeHl5xvH4GX/561/w+LBH16ex5w7EKSUJ0mHMkAyuRA5MsilOI848U4pmAjonaS/AHcZwhGdZgQTAO0LnSI7bjCd4eOw7h/5hj27o4XyHOAb875cDnl++4vn1AHgP5z3iywtG8ggh4vX1BV+/PicfiZwBqGclnE6nfOYzUTk/QfcIvDw/4+XlGa8vLxka886BEMFxxPH1Bbu+k/TYzgGj7gkZEU4jAtTvY5zS3iHqju5O3lUcT0Bk7PohWcDA436P3bBH1w0ASNK/cMh+NNBaj8I72dF8K2q2UTyLq+7fvnDXn44GTLXJFsOYRpS0GVQ7bPP+voNZ/TficW1hd+EeTF9ril9Kv6WIlPS7FdszTTT/S6u7c8ki+F7vRyPKJk+xAsoMmDAKFm0XyfHpHEaa58bx3oH6DiH04F1IO4wJow8YFfIgEQohRjkECSLMj6eTQFRjQIxpl3D6j5ll5/S0AxlWKs2Xt+0g2UXlAJ+IMci5BZEjEERNSFgWGBJOOo6jpLxAShme3sWYHOWU/sSX8SxtNucxx8j5rGa1nMq5HZS1fBWkp9NR0nMzi2O58wJdedmZLYnvSgZUxJiTBuqfzCsHpOyxhJKWxe51IBLrSI0ubRdYYbGQQrKTdN3gM/vulsI1GtR1C2ztc5acvhcgkTe358eG3LbTnCEsl5mOc/5k/AwZVKLkqITses4MBuXchu9tVd2cmlqgUSqMqWChMvmpQCAKCZVqJbmgSzASYsw5fY4uwDvxR5BT5hgRYsnTo+Gd9tB7bU5+hvl/aar4GEqUnPw+9H06hS3CnUYEDQflMTPKmLTzEEacjkdxtLLuXeHsGB5HyQxLSZidTpp51zJnzierOefw8PCQ0ol76DkgVjBoVBFQ0og8Pj4CLFlWvRfYTXurkJCFoeREt6GMBxc41VzMG9PUsZPHNsFcTMgp58mXtB1r6LsLhW9LF5gBseU4l8tfvRt2TbTRH4xxWbJDtthN+0MZ52g1f6RDSxjw6TN4BNwxFQjwkBC+I0eE6BPU1CXjvtGeH5CaqklmAFM4j4my9UQQ6OkYRoF/oqTVVlBb/QVgCf31YATv0HtC544Ynewb6R0hRkJkB/hdCVkFcAwjunHE8bTDAEmT7nPLCEwxMzZqdyQpuoSH3T7vpxjHEzjKpjBOKbSRfEeRCJ7TTBl7RO8QOOJ0eJWwUY4YY4DjHgiM0+ur7CVA8V3ooT2cch5pHiHdsFaUjYjOEYbOY/QOoyO8HF7xMp5wfHnB6/MLnh4fsNtJhNOQUmzoRsJkF8BTmY+y/yIpMLFkR42uROV1XSdnRI8nxCQkWPNVOQcXSyJBYj0E6TsmxLu3JratLfXN5wpf+H1W962snD+WAHjT+1n3BAhLadZoYCHJq8MMOBbH8x/NSBBq7ZdRy2kZguSUT4iIEBg46X6H9Oe9B4Wyt0PyEzkAgs9n6IOT34cZQ3LgRo6ASTsSU5SNSzBWjoqpN+3T/LPaQT5BJl06SzkLn6AhBZyazqAQwV1KH+FOEjKboKG8ryEx4hhjPjNBE8xluKgfMAySeVUtDOvjYr2XBN7i/T7BOBI9pQcRKWxmD4Hi4CY+M5sRtrYmAKQzyykHEAh8xPmchBBG0JjO0QaDk/VAFCEZD77zeQrvy0TfooXzwucW3VvFbEm02vjmhbLn6Fu2e82ztpaf31WYIAqs5NJZzyjaF2LMTmkC0rk+NBvZPxMJNh8QdFxI8iB57xGJkEAn2ccAD2aUg+pjhEupKAgs4ZeOJJY/JbqbwlUJ33YOiA6gxp4UO6Wzf0HaoGdIl/OlgRGjCCgNFlBYJu3MjiGA4XI6i5h9PMJgY+R0FCmgp7z5dArbbv+APp2jAJTzEcbxlAWQpN92KcNqOk0u+wtSgjvSk9ySb4PMWSLQ6w6s6cN1GBKMJZ819k6d6w7RS/JAgY0iQALpQfd4kOSU2uLX+uMLhQkkZOkShLMGPgLux2DXPF9XzTsZ60wbmPzk/WycYxYoz4x+RHK/wcdXOYAHAHGEZ58FQkzwUUCf6vjBMaQ3kkQPBYxRnMfeCSbO3ZCglQAQyf4Akhh6OJKMp/6UcHhgz4SgTlLvsBskTbTrPMg72VHnRNgQcc7COuUvFiK0+YaSxdJ12O12WbumZAGIo1sPnAFcjODTCcF1IKSjRtPZDd57BBACp42QLE7aCEbf7fCwf8B+v8fj06ec4VRx/RgjxpNAlM45dL3Hbv8IL8MhR28eDjgdTzgej3gc9uhTKKtXAQiCJ5/ycDEiYj75zycntkBw6pCXfTaCBsopg106TMolc+s4nkBRDqDqmCVFOHkMfht/+CHho83133G933NTX+Np+OEB8BZdLdfmuLk9UdM6pCdiSp2YLJvX3ptI/RYUo4jFAiUJnBQTtu5dgii8B/c9TgBAMcM9xH3WfDlH4kgIp9OTxpL23Ce4p08HJCk4Inh3K80FMHE+M+fNby7lVFKYh5kxQBhxx4yO1SlLcOSz4xUYjQNZ6okp+CAmbVzhHTk/+RH7/T5bCfqsvOkOJUOqCicVChoFpJZA57uSwZfK3gs2ZzJP8xKJoNR04PWIWOBUnNc+p7mIaSx2shU616ttXEPvwtG8lbFuFQrU9mAttWbltWvoHNzzR2RNtxVg5bXXYziVBOUY0wgggBgJRpLgUw1ZZQDBaKE/LC04aFtnfAu6ljBqSDy8/GAFJxvm6cHew6UoH4nzlfh5Bw/PLE5dktPwIhOIxKHvfJdSkHSZiZdkcCxQDbvyXo2Mn+0QTwalwi/KSCMzvPaHGT75NMBy6A8gZyxEu+WdSsZTIoCcRFl5V9Jq73Y72WyWcP4pU2VA+6wOcqOwFf+LB2uoaLqu0NDke3U95v4XoZD3G1T3spOIq5jCgUOCt3w/wIeulHXTyLJz9C6Ewha6TiDku9c8YaGi+XV5b+vaI+3QKJo1AuHHFBIE5NO95jS/rrHXaylOBMHcLNDYcwAgOmUB0TvJjQ8CIkWEVNfRpe1TxiXxo5ErnMmQeFJmHWrM2aX0KJqC2bkhHWQ0IhLkMPlUtxzBGZAGVOL+HQAIBNV3nfwZXF5DMMWhI7mRpuu6vNuChet3xcndRDBkxutdEgoAx+QDiMIsiaPIygR7iSAAejA6x3IGwm6HT58+ZSshGE+47qjmlJHVCoIQTgA5gXjSVU8O1EkYLYCJE9mGmBLJjvyJUzlFgynbYeRHJYEgaykmH5rslThiDCKs+mGPvk85pjoP33fwfY819MMJhXdFvB7O+X7J6f7ExLJ/Yb6hTOCjdXrTj0CtedW2FEDIUSzCT2KOOCpaq2jGIQQ5eyBlBM3jqFpzShHB6YCjMaW11vxAne9zRM2kCcrU2YESpDNBAHLbauHVqCNF76g277oUrx+R332U49rMaWZSUd/36JPvpAuMoZNzIXbpnGdJgDfm55WoowRXJT+BOr09lXOZeQyIJPs0hq6knIBGLKWdzApp5QywzHAdTQTE7BUazf8YxrSrXJLohcg5KmkyTjOIaplWC4XIZrcjLzG3omWUDl1ihFeY75t9FiqVV9zXLHIjZj4xi3n22+xr1c8WdHbOcjorhC4OhV2BdVvm1lc2fqn+/fzjcp+olOTZHSYl8qSidq1lL4MeEk8An0AkESY9O/gUeQI+pVBVwgs+o8zaWPqkPSRK5+Waxxv4pdbWKdVAoInGuYbWKBDTudQovzRWBj6bAppqycZ8kV0HeAfqOtDoQJFyrn+AQU4co7J3vMtOUREK6Yxll7D2STiRbbb6d+x702LW6Zx+S9PFkaa3cJOx8ESI6SAlpnR8qxy8IFlhwXAxYnAE3yfGDieWwn6PYehAYITxaKAmZCUDHJNS4eBJhKZ3XT5b2REhCsoG74HO7E3gEMBBrACOLOkoomZGECvkpPtGUJQacil5I+ludBE4L4cDXg9yelw4jWJJOwfEKFYz5EhUB6SjUS/ThiypJx1zLDPycq5ouRELZQsR3KpFIHVvPCWLRywxj3k7gBbEIfW0rsXVdQOilbX62ep5i9lfgs4ujWH5fW0eFJ78Y6/XfpppBMlcI2Es6xITvJRknKb3BtOO80qEPMe45FL+H8JL8i04eH7JdXwK8vwIh/+JJyQsAzDPLKGSrmL8KfOOmwoEsjNJh3CjIjM7eEZ7ZOAF/bdd83QRUvWu1InpxAkj4jONe2SCQwTQgR2JT2EQjDpyQBhPYD6VEEsnUUnsHTTQhUjOENC0GA6c4BsqlgBpyzg7/tPNaRxLnyfWBLPk10saM1zM4aEhRnkWMdhx7n3nGYSAAcCeAMcBewKGzssmsGGA7yTUdb/zOLwKo+13D/BdB0dAjKIYc2Txp7gOne8xdLvyjqJmd5Vx7To5q4IAHMcjYogiCALAkcARCKOkHJHA6YjX0zGdQJdAQCeb/rp09kSIjFMY8fr6ii9fn/HyesjJ95xzgA+gGIEo/3pm+PTvGvqAjz7ou9I1qX3vRUuRSOegP8V+QWlXqZh4GWa4F2BIlRCab+VTH5aWuGypKNMVvl2gIoUeuq6T3cMhAKMoW5o6RO+3m7PWPI/IlQ1l1W+RWY6lTPND54qmqtA/oAgOroSfc052OEPOf+76E7pQ7nPOYbfbYf/4IOdIdB0Ozy/4+vUrfv/9d3z+C6VDeNzE/6X7OGzmVliBbSbS8XhETInzOKXcDiEgjiHDRrk/HBEQsvtR05grBCRQk5yH8fz8jNOpCAOFsh4eHvD0+QmPj3K+Qt/3MwjvHG0QCrZSWvFZL7mLkMx7wdqvSYh3L7omyupHpowirCv5ht+XSA7wiXpmtDFrmHyGOixgw/YbkwiHdB9T0c5VC177Ns+tB+sIzp+Ret2Ci6iUmIB+MzguAV2CUSBvMqOy25a7HmDN0eORd0SndhHIGInUFA6lxZhiV1WftGH2KMpcLqbjVOsBzVZeEQiAQEeOKSe08z7lZ0plhmGQxHWJeX4df0+J8F6x/yTnMdR9sFh9UWxmLwHMjNMoZzgcTycRCiwJ+1iFAZdstpJlOA0Mqecr+TFSLqnX41Ego+MBYzAn2TkH533a8T2gHwb0g/hGvK+t3GVav08B07TD1RA1yheootkY1vvScGxYy++FWbYmwRJdI/jei7C8N2n8ttAZn0mGo+8jGAYcciqMkSUFMYPEjZCYZVTnLRFKaEhipLqyUzc0jfiSBXKOJkzfRtfMnOaEoOFTBLOUGE1H84xU2TN5vNL70ETlWfuPPQh6xnMnuXtI7lUBMNVIm8DoVDhVn7LFwABSgrocrWTGhmNqXX4d0nlHLh9tmfcWRPFryJ6JE7pTSPsx5D4VCj5FR53GEcfjEcfjIWvxM2uRi2Dw6dQ2cihpO6ik0BiDOIKPp5P4FLQ/EntaoaKU9mTo2HuAU64klkOAXg8qFI6pXIpqSqHCEmkkByR1fY+u71Zbb8AHfPTN6CP6aD21MPT0y92fm1UZnqfTvvYs7a1KjPUXtARDTUZBzkrmkpCdPGODLuacQ7cb8rGnRx7hRpIEbOyyf0F1R3WcZidtuxWlX2QOn6nOnFa4xY6F/bdEBQl8ZCGdvL8gharuhhMOJ0l1DUK2FLwvaSxenp9zFJUzwk7zHzEzEKbpOYZBsrFy1CylEZFkfEoG1YA4loAdC7vpGBOAYNJfACRRXUEshK9fv+I0jnnntvd9ytpK4r9IEVHODLqOwe33KfBShQ34aOZVpOWyIADrncFJhVhZVsv/iHQLBtjCmb8FrX8/ytAIMmUsvMCsUSPKxDJQcnMiMHbxOWHWhctGOJxyDnwnZz4rHGTm+AQqypQAgQlEc5mYeRICOtv8VC14MlaCXLMdq545EQZshIlsOrPPEcenkHMOLiUQ6fqIOIqWGhlAUKhDomS0D1MqlkgJFIoAR2G8XvMQxaT+y5/IClnzMSqmz0XukdmAB8otZiPUHREYgrf7BCEJzm5TSnM+P+H1IBaCT+Xt5jWNIiLnEidKzu1OIpYobQJkorQNg+SscJI9BezEbKT0r0QeaT6lgMgBI51yfxjAGEaM6RjOw/GIEMWvovstyKdcTcOAYTdgSFaPTae9RTHZEH20xmSnxuelMqlehY9WNvq9QEffht7KAL+XUNhGU6ucm9fzb/d6/Qz0OCD5NhHQJWHlMbqCCWn4LCtcBCMQEnNhLp9Lg7eNfb25yQqGnDLBEG18BKUlVywQmgmf1NFchsjDMSdm2ScmLsn09NGSlXZuY6nUmkYmyiYwhW3UOZ95gv0jIPJ0oyNlp/ZsJMAsskra5KAnqHnjNLfjyIzJYTnMcg50Z3Y0O1eymjrnJE03MwJH9PoCKDF8cjIvnANc0pFdkgyU2s6AHAgEHNNegxBGnHDMo8bgfCbF6RRwGoOEqxJkN7gMnERO9X32I3RdaXd2wn/76COjxW3W6H5MRr/FGfwBHb2diNxa3eHNz1GNf/reihYsUMU0bDTDGShRPNeQpmi2dds/xcyXBMSFzlUnnk1hmJoUvpFDciTipu97UNxlDT3wmHYoJx9KZsxT00UZaoZ/olgKcnDNcj/0ukb/WLjIQi+2PAPZYGXyAPl8QI73ciIa+sFkPxV453Q84nQ8gYhyvqZyLjPnNvRpd7BadeM4isVjXWAGelK/AzqT+yiUjWuHtNfgcDzgdXyeJMQrCkJyaANyBnmMIqiTxdMPkqLj4eEh7cgesl9FfSNraIOlsLQaW4sGxuNyrrxeWr/S72kpbGXcH36C25As4jn8UqdfEA38Ei5arNFtc4Xhw0meTwSmV4CAkAAJ5ogRHjHs0neHyD5BGISgTBs6j7JY0eqxRfmZRNtw2Y3bnG9nzITZVTsmCiWRbLVwUeEoyqNoj4vktFPZ+x6u30EjY47jCZpwTpC1VG+JiC3JcDkm6E0OmYlxzP6CbKEkbD7GiDgGxBjAISCcRFA6R/k0MaSxqQUnK0QDeYeRy6Y7IsJ+vwd7EQrMckTn6XTC4XiET6k5Hh8fsUvpLrz34iw+HoUB7wbJbwTkc5oVy+f0zmKICCypuiM4QU2SHoOI8rGip3DCIZzwOh5xGI/4kg79iUbDdxMLx0N3nff9gG7o0Q8l6eAwDPCdRB2Rk6gja7NeojcKhXrKpTJnoaZKS8mQwPuwFrYw+Xtqrdl8X9WO9zF2byNSgKFcqRzO5esaKHPbuBAAypsLCZ4Kfu1pRGTZFAXuElPzySCWlMY5YmSiwdJkWq+dWWs2KU79C9Ss/JyulTVOteqpiNMs0rhYClqekLRy36XfAkaX0kqn3EK2nZP2aN9SKGtO95CEQk7FwJj8FjV0M5WLXKAcXz0jWwxUsrEyZCc2cn/l5DJOifoYjJjCPcdxRJcsiYeHBwxDn88/PiRfg1pr0tbiCGdmcHp+PqYzhZzmcxjCWCyldI70GEJJZhcjTmkjmoV8GAlySmc9+JR5ddjtMOwk/LSE3Ho473L5rfRG+OhWzOh9MbW1guG9WQlzzbr9/RLsVferLtuqO5dTz/Gs0oXLtq4zw6na3xKV9iiWP297sUikMXU0i318gYhlR661ezlpusyUBQO4QAVTeKM0xzp1z1HTmVzVrfHxqcBsBWXjIf9goZwNFsvk3GLROglIOYbkN+cdMNa1FkFT+4J0vEMURmw3kvV9D5fgt9PphHAaJ9aKlmPDnKkRBOMcASl1dmQbnSTkvQfUGcvpDITjEcfjEcMw4NOnT/j800/Y7x/yc1RoEElEUD4jGcDpdJok/LMRR6eT1H04HHA6nQqME4tAsaGv8poKfGlhH4G+Bux2DxgGcSr3SXBpEj+Fk2w9W+gGPoXWBLPXzmu8Mnnvx1wJ6xfB1gFc4n/fg2rGfe57i+nUZWuGufT7vH5uQjxLbqbps9plbDvPBTy0+lzDMOVBVd0g9LndDIdDvs44oXdyIE+fslkegsfrKCGQIUUlibChPJ8pB9EjpbOc9mWp/boz1ZZp7RTW8pHLjmYy5oHLPZA+5b5zyscDSXNhgS6iBAoRw3sHclL3GBh+ZHDH4nwFg5yH9z12u0/wfhCH6HgoVgOXVCGcIodym2PRnk+nE8ApSijESeI2ZbCn02kKo0XF1OWeyBDhVCV9G8cRAR7l5DVpR9d1QIJXJKHcETFGDMOA/dNnPOz32O92kwieruvSUaOM4/GIfTp6k7zD6XDAGEKGrGIICGPAy8sLvnz5IofuZGdyyJaJnI1dxmEcA7pulyOtuq7DbreH0xTZIWKMjJ4jPj09iSAYBDKSU+OSpcVl/NSv8B0czYY28VbhBPdUutdm+lHaBDvcqeHvRdhspivanS2MBk2ZoBa7xmcweSLqSWqzSkouHs7XHYf04Cib1wgIzPBO35MIgimko+3XMtNntgSrFcTW8TqPCqqpKF52KPWJ1obKpVNd3vhyilAQC8c7B5DLQj7bUxnapHQWgTAj5zidqCbCOzd14T0RbH8kMkkZvY5DbSXUexRyOdbEcvYc4yR8kE5YSym185ja56T7uq7DLh0MpBCNtVCccwJ9Jdgoh6mq9ZhCnlQonE6n7ERWq0iF4TEJuhK4AJAjdNQhBGmb913aR+HBJELOe5+gIYLzLkNGRD4d7OOm47Jxnaz3KWxc7fN47Qv1vwNGvJXKYr8HbbVablO+xaguvZva5G0Vt2eczJ+JzMFq+VAgl7NNMJDN9DmzfircszC+GnIKhTwyh2WU+Hmp13uNBnEAFaezQix6n3SPJgythvPqf2sr4ULvU3mL0Uw7bcRU3vshYe7FSnCJmaTwd3luclQSiQAgkrOECcX/kgVEC3/LmHjDOiPK+YNiiBmf1/5nTN7AKpNeJYaX4ZUkBGS8uDBvDggxHb4Dno0nJxNVI6vKaWslhYW2V/ov73cSnsolwmccxwwNKSR1OBxwPB7hO9HkrR+iHgv9rFp+n3Yl++QUt9YWUOaHhNz6aYpsI/DcpUWUaIOlsGXDmF10P6rK++PQGsb9rWhJTjKjGX9QWwJt7n+9wNsyLnpwvaisbLRmiUryiFDvgnMDnNun+/YCUURGQNIYpSLT52kfdEEX38NUIKwR8uW+YoXQhP0XX4m1EspzVIaVaCNKgsKrzyI5LLuug3cpsRo5kBtAkBPO4I6AEyac2z572Vz+GAAHOAJ63wEdY0SBdmyo5ul4bKaZIAYQ0jhruCcAThlTQUAkqeM4Mg5jFAgKxaHNrhyK45yD7zx2ww775LTVMs09IwyMxxPQMajvRAgky+Dr168JdgNejwccTyecwojAEaeD+CQ679ENfbYc+q5PRhjl+SERTCHtg5CIKRupNY4jCHJuhRwY5EvIsCpoeWPdek78kebC0HbGel9GvFb739rut1oJS/erxrWFh5d7KuFQPWvJSV3qsNfOwEuEtnRqlNcmEev/CmbvCPDaUUoigyLATkRHjJlVZ4gJVUQJ80wInPM1LL4L54vGOb0R9qIVCBI+Wc4UllGT9ngqjmxyPu2YTZk6ySWIgpMFUeCK88Szz2QYsnXOa1rvlvM1j4+FjChmy6JAJul0PY36GeVfUJf7Fg2E1KXNX5ovyCdIzG6XsxYGM+e9JLndyUp4fX2F814EA3OOPtI/IkI02U+lXWUcFD608E80/iYtpyHAkqa8m/xmQ5it1bqGNqS5WF1SaMPeg40V48P6qOk2PplaG6vhjZW1tF/PmdeW8+krfGTumQiIhSfO27fMYItSXcNKBRpaoonGnbRpIQc5EwSSIz/h6RMsP/1vFmpoGIBlems3Gsl9CXNTXHqh7dpuRy7H1Dsqp7BRglYclXOQnfdwnZ+kbxaGU6yCmQN8MqZVe1ttq/rtgJJDKL+X9rMkvXYsabXV9yAtNM7sZImYncwSril1+QQZdWmPgjJVQbWmSod+1zMMrHKS9yx4GTttU4ngKu9dIaK8U1r3OUQGE0/mivVhZKEW02E/VAITVGjXmwHv5GjeynXuxLhpadr/0ciyoLXl34OwXGj3udeW/ACqUGsEx7Q72yyQq0h3WC1APikTEgCJ7OkowpFDgMJGjAN6BHYIabObdkE1zvNWFiaJ39Y3u5w15yohqn4AouK87r0yRpTrqfuKabvENL2XP8G5O7EKnINLuRsIDCZn/kTI635vYYTmzIMYc5irhPXqXzo1mwNCVH9AyHtHyBE6V4SXvg+dbyIMHGIkxCglAlxKWxFy5FHXTfsIJ/i75kPqU76jgsWXd5bPfUhaumX0eRNcijDCKYXYDsPcJ5GsBIv9H8cRICoHRLH0dVBh5TziifH58ZNEvI0BvaP0Hhidm57t4JKwyBYF1i+fOwkFvu8C3tiUH5W2oEK3gL5quOha+GoLU7OhrKpNx9iAC7DW6Wxw51bBDB/R7DrH+eVGg9M/aQOWAzg5oDNGn5y5jiVhXNJzJxFOOtZ27GzkzdoxzNpz1XvLgETjL9ZNb84Mdk59J5Qzgk7vlb5QhorS5wUrstZsOUn76ffye4wFImI7ttZBaqCW7ITOY4kEY5UxBRiRRWTYZ9jxkL47UKo31+9cgaaYUU6T5MkmuzoiyoZ9Wuug6zp03oPTWQ02rbfV5gdTp6RzKT6n3C4UiMmRy/CVrdP6qCY+GGAlzHeTNBdzog1KK5v/r6v7ftJma93M1zDj1bXjfhJtbkq2BEKt4SxFgMyvNZ5I7eulRfWHOZ17PdN3V8z52TtlrahmZme0eFuazQ2cnJoIUFuAoNkyRRCQwcAshm41ztKG8wJ56Rppl2AdxkXTFydqgX6Gfsj3Oie9U8ioFgq5L2nMCrOh5gspIr4wf8tk50Khii6qBIgKoulmvfIggvSzwI+pblATy9ckesycIbKu6+B8gciycEJZBxa+yb4MvR4i4hAn79n2wfsuj5WOXy3kHBGOx6N5p6WTpP2sBL1tV35fRLN5v1XRuIuj+dwCW7hjQ93vJ9/Q3eTBd6CWP+Gm47wIH1GjTMs6WNuW6WJYK7TlsSmnf9UkJppFVVGMAI6yKc1FEB1A5OQ4ARpAzIj8ALBH4ACQpIevrYQ6PNJqo4ttrRa8R9JnE5+22L/CFNYRWWcIVXhGGZdCR4CcZUzl5Jh8j6os+otl+DFBRKTXU0qLzEwNo86J2iKng+aRGa62ZerL0LlpGKy2i6Q9ssEMGJlS/W0nbUclDBVJeCorVsGqz9PoJu8lpUc4naSvaTfy2HUIY8hQkj1icxgGRI55j0FuA4AuvR9NCijnP8u7iMzgEBEoiN9gt8vWh0YYTYRYZVnpO1GY6uYJ8bbT2kX8B+KsN6TiYLtL7bnuWhAo1ZaC/fcyo72VMKkdZRI2X9dfC4yz7czwUX1d2AEZxLrVDb0UmRHSQSsRgqd7b5l16xFz68tCEFZjrhdwSwPU+8hiyYbxaVhlLRRq6EIYKuXO2booOTO19xaqapGFA+34Z0sBPPmujFNFINX1mIdZBgjDALOwsP/nUk8WMFWUk55/3HVdhqPyzQrRMHIaCuvgPZ0kk2owOZOsxq6MWKwCB08e+/1+urfAzFN1FAcXEFLYqquy8E7gQOfy2ctElAWOLVtDlN/JUjAPt1jxGZI1uob5kfn/UvlWp6sc7peeskE7nnTzLtSqfKl9W6ytVBO1GWetYXOlJte31N+vEQmZV2tdRO0uab4h29ZchzC38nthYM2qWm04V6AqK3KEQZElxWgEiE55KXd0QGQHTxEHcsIQCZim6lCoAQhBcXQGoPHoUipbDjyVadL/5LUwsIRaBC2hYKEGrQMo9YrVpL8Zi0Xh0jTxp4xeYaE8OnLVCALlC0UwJCdzjHOLw/QP+o6zlcIyhurcdj7Vq89ziASE6BDYg0HCY5yk0AY5uK6D7zx8J8njcr/ygIuQDMcxZXMNEqaahKIqBJrEbkxnLxMIne8QfYkMohQNtBt24qw2Y2whLfIETw6HU5yMhX1PtdC31kIthK+1/Nef0Uy+cdVqs8bsnmi5VuMxm0CslW5fxvlWnPFBLNy/UePeyuTXDvSc+Sr6u1TvuutFG1jVjNyWZOkvlZh9X+5ndZ2prYnjXJ8MPHOOiwOIsRpzo+WaJZHrtWpES3Ot25EXoP6xfVOMEmpN6BLDDGBwHNP0/g1IOPievgBeUm//C3/DyOJED+gQo76DDiFoLL3CFAxHx4R1l53TlMWNBPEiCY2eGM5zhhXKyWLFKtDvVhhctPgYEkOkgoHHLAxgIKCYmKYweICQcibFE0IUrD2fWxBGjDGI4OMIxJDuN1ZSyraqm+fgHdgRRifjF5kkG6nv0l+f32cIAewdQnB45Q5H7hCcpCdxXQfuenDXY//0GbtPj+gH2fg1UTTSvyFGHA4vOB5fEcYTPj08wjnCeGIcx5KQLwTGazqDASDs94/wvs+Qk4OHg8PDbphYR1Y4H49HdF0P6gjjoWRStdq+WhT6p9CQjTpasjS30MfmtT8hvRefzJ+JyBE8JD8QogeRHP+oOW4mxHxOX0gVTmEeZQ5DlULZMp7aMqgd3VvIMqoMmaQzglVQ5BBOK+1NO2w9LYoxiu8mlY9pX0CMDPKa8qGc36z1CNzDiLFs4HJJs+666cE5eQyJJoqDMu+8VyLdr2VbkOrEQU8ls2rtVG69E9sW9THUsFe9kW8rLLSW1lsKi2kuuPnR3mkwgYU7l7XLaT3n6JzWcx8muCkf1BVQ0z1Z99aJtD7KirCUEiXXYM823jyhW0y0dbkkRSsPb83FS/3ihc/lq320oxKjL+MghTo+ISDkORAQITH+EOiJGdEp7KJRP/WmsJLDyJETbZpcPrC9/ptEt1RMCWhDDK3PShbvr7FqgTDKhi1GSdVgBVjLI1E/S5hzeiaznI+KlD5EBZFL+yBSYUo3MiNBOsi7ykFlM54KT2HUOh6ld1LPNLRU+yhoCWdL0LbdMvDaKV7vMrZjmJ3lJhur1fxbVAv4ep68lVYLBb+KARpTPbfRrliqi5ov6zs1RYrPLFqgmL5voMxPZj+URX+xjnwq/cp+bhQg96RtkUhtuG6ZxWwlczay9R/wVHPLz+X6+VpOrl4UdsptGj9oHn+NFtEHMgIkhFA2PjCAnl/gIREljgd4ZhGdTk7kcoSUroJmDEN9P4qGieZbLITdsMuHwdg/yzDqv9wLw9xaYzHB9801q7XKucwOzHKmcUwQWN2PVt01UwWE0UvEEKWzpBPkYvyDLgIu7TfhyNCzoeWQmoBTZIzROF6Tj6Xvu5mlIM7zZI0Y30R9hoPTw4Qw74+Gq6pVYPcMaN+sIxhAPptBnddqHfR9j8PhMHlH9fOcc5M8SLekD/jog354En/EdSG003vuJ4m99yIoIoOjA9M0rt0RoXNlOdoonpzYLjEA2Xk7ZCjk08M+hzYqU8r1Vo5lCxnVAQWXiI0jilmZusA7Gs4Z9TkoUEgrO2cdhWWv2b5HZlDanBXMpkYwl3DTxFSVkcs5BV0WoN57+E5SUO92e/Q7SY1NsqNv1rbcngr777oOh8MBHKdjZs+O1ndgI4PswTpWMNSWlgoFhf6sUKqhpzpooE5r8Ra6mVBY0tc1cfDtal/z1CndZKia1be14qU2MC+NRbuFlRtUNEZ7hUpUBvP8juXGlEV7H1pqRwuy2dIOHb/KUVoxlbWLo4ZmbMuaoGfDGVl/diQb2pzueUj39DikqBiFUxiOGCOnYy1BAKlDmSQRHQEu5VTSRe/JyaEqfRIKw4AhnSHcwqq1ny2fwhaBYPvPk+ANKlFfVDBSZZBq1bSeZ9uV7hKICGX3NzMAkp3hkfUeJzvFYwQojVeypkJI96tF5X3J35RSWHSdOJcxCQvV9hVUg5ESAiYICCnzaUTZwayM3DJ9y7zVx2L733pHNdVMfmo5lvrr8OJb0E0thbZbwJjWb6t9UuXZ31dc3kLLrV8vFGS6NcHv5jVujJusD5p9V9z2fUBTS2NSQ4dFG1weky2Q1QpGd3YeptPLRHojGvx+wvSrFk6aoB8obcRKzWKIUNAZEEkS0QUCPI8ZFhEIRjBr70pW0mwFOIfO+Xw4e4428j6fhVBjzdPuJ4hko3PZau61Y8yRjFntG7dCQe7TRHVTC6HAJ5oNNI2RXe6sQloEApOTRHhBk/lxFgqnUTauwTnJ7toVgSCJ/br8FwlJoSpzVuohaJJDRx45EopKeg11Buv4qEWhUUHWqhvNmctLPoAaxqutB2AuTGqLoX5f10JLPwh8dAtr44O+BS1ZILWVY+94M1VypSkYLiyOe8+wGARmSUh50v4BUuuRhWGq70AZytCLVbDb7dCbQ1TsfgQYhnzOh1ALg82WgqGs4deKi2Foru/lWgySCmLihygMTbJ9FujI4vAhRDhXYCgRAgn7T5q9o6km7r0H+R7oh5zCwnV6gE5xwsMh+w/ac6bsD7GOYwZyHfVZC7a8FQxElHMVWeeztTKUWrvaLTRlBYENSb2VxbA+99HCsyZDeaE9PClT7nSLxrilhsOYL/yuhS6N05WLY3HP3bSj0G/L5ws3ri8y0RsQLTPvW1R+eV5e9+zymqbPKMjZ8rOtc7puSxFYjRGvLzWRL2Md6bunfFZnMuIoPSvBKYkPeCLASRt0X6/reuz26WD2ZBUMwwDvHLxJi5ytAm1OA14o/efZ53PhoE2aYpcV9JP2F8AyR5cjgGxyvPwOKEFmoJSriBGi/KnlIKfESYpr8h3U2gqs5y6I05lEkkqCO+/g/AAadoWJegdKG9fgKPEzEcjC3E03eeok1z6lUDE4ctkC0I2BgElWV20MtJFF1v9Q+3T0ms3GWsOAdle6JsK7tbN5vaXQeKhZo8u31eVbAC3YMKnzjHOGSzafZAtcEArJ7r1qSFvrKXey1qD0ylr46H3RlknXKjoHiaj8cAUxL7Wp3U4VHPUiO9fCdiWXyPQrabVm2af/JNqIGSks0iEyAU40664fsN/vM0Q0pDODNZPp4pMrgVCPT62F67WluurfbUhsbVkxc3bAElHKnZQ0abYCyEIoLgsFZtH+VTCoUMgZbZngyWfNPjBnRi4+HBlZl5zK1PWya7nWopMQmysBhvkjzIQCUhspQVj2uEz9rGUnx2GipOfIVtHCO7RCob5mhUydRsMK/1vQu4CPLgsEU5a2RUx80Jy2CsBtAmFhwmPKHP9sJBqzQA+BNGtmOvYywx1Ponn2O+wePs3CGyVDdyMNRPXvJc2xFQa6RGud0gqrKIOKcSpMyv3JOzBrn0mbEQukEs3mtV53NJuoHLkvgrlo0V3XwXU9XIJtlsZETzObXa9SeQvjLWX0GcroVWtXssnotDyADB9pnXWbbCK7WnlpvV+7H+KWtFooHKIWrU10PmNSJwtRCyTNSSooUQOII4oJPtWrZmRfFtrQk2KyABCdly1EkZEiw83ApqdETYHMuW+cnoX8b0O7onRyk10w1de8wY0AyWdpu8K5VPl/uhI5R1As0fUC8vaxzUqLexoqyCqXMZBaJGpEgTQquvB8+4x60Sw5+CiZi+LoJTgu7w3mPcXmeFNmXAwnMCEXuJCZcfI7sB5d2T3AeY/Od+j6z2IF+B5ueMzx8M4P0zlaGjzrg54AZ6+fcybbxHJLewWsH0Jxb5+6qvcTlQ1es8gvNsdpxpImXDKmhpTJlAGSUNPAhMiE0xgwMsAksEy/f4BLjuFAHiMHjJExplfjnMdueJCD7VMoaIwR8B26ocd4ksNrnOvQ9QOIZC9ICAzX2Z3GNg21h5dTlOA4QVdUoCBrRcQYcTgccDwe83XrL9BwVIV6dJxtRBIz54R6OmY6jjZdSf3eWwKhtgTv6mg+jmVhTB80NYdIGX66JMf9pd+kQJripNwbjHG+1AtQPCNrV0xEgj7Hto8k1zqT2WiWzdt0v/PItqhpq9V1bFMKM04vvxYYZD6bFnLdNntPfW3h+q1oa923FCJUeKhOiuyArUpWExz2phXPWYKKzlg0RBn6AXOes/pe9S7Fx0vDjFBIn6NurmIG3ABKEUU0POSUCzR8krj0rgd1+wyrNHONLXT7Gky5tgDOzYf2b0az59KGuqRYENP1Yc9QYBTYSP6Suug8fNfDdX2KFvIIgRAREVIZEKUoIynbpZBTHseiRVOBs3RfQox63oVDOa/B7F5OEJFzgNXjrHZud0e3FA9rHbR8N7XQbh3XCWDmWK6thkt0DQ9ZLRR+//333JCppq2LlfKi0YM9AMlnpWNWpF2RvKLtzxO6ZaY861OZYpZx28+cFjOlyAwQ5+cCRcISTdV6BhIuWiyRXLvxKpPeqzhjEoRJycwyhEy95SHTei4h2fVLzRqZ0fasFrmOrA/nPZOx3KzRVk+KyjrLd1urhc28aVlYBLSiKYyMT9p/qbu8C2MpcEHeJRWFMIh+v08MzsPtn7JQQP9J2kgOkUq2y8W3Yy3ZK2hxPl2wSGWtCNYeQpBEdknzJ0cgFps9MDeZoD7LWhbCCNM1oyE7EkhNtGTdGVyEh75JZZbWKeucwzAMySndTjtR90sywS5j/XZ8rEAAMMlA21JCFFqy1oW1FLQNVii0FBiFrFp5lG5Nq4XC//9fyEw/zXUR1GRM8GxiEbyHSFsqpzp5L9kT1bmm5XeOMsRU9Cw16ZV92ciGJFQAiQdXAZUYtw1ds45mUrCJyCS7JDCHdFB2RM7xNOXksNkxlZx4LicwkCiQEqee78if5yb90it1ziGcSdExhag2yIN3TZwNyHrX6KRMfeWSZK3uYm5EKdnsrqqJZu1f4Uer8TrZy0AEkIeqlNH1IBIm4bsH0XK7Dt3nv4LS2chdv5MwSeeBri9OcLiExfNFrs/V5y2vf4tPQUmEQtpcxoygGjD0UBgU7c88p4an7OfIwBiBMbD8GwGQh+92cH0PP+wQXS8b+5glRbYjwKd2OI9IXuAm50FdD3LCOOE94Dq4XuE4n07/M/mjnId3HZwjjGNIgyoCooxtYrzJKoFjOC/HZyrU06fNg1bZXPLxWIXECmQVFNbJbMk5l0JqpxsV70GrhcJvz4eJtMxavkNhyuAkScu5rj4JBiJKRwJOoyOIgNGPedejN/AUGeZOyvAdoeuS5x1JiKiiRube1O5ZIj+1DiZMNS14SrEhqh4210xhDXlBABPBUKycqVWwiYywnbVgor7qU8v/L1e9zRdxL//D5BmYoDaNNrSvn6OZxpr7USAPU1oYTy5+Hl4xySeSlpTmcxIKXd+j7x8F1uh7+N2nfBiO832GNFhDV9Pco2yxtCHFyXhsG47JuFyiFkShrVLISOHRljVWN7zG0IUJJmvBvieiFDXUwfku+WiETRM5SYTH4heQyKXkMKbCvMmlqCaixPSTA7fqnyO7+StZCnUHlLnQ1LFLXCyBOn21fUatzVvLoAUptXxByi9rK+FetFoofPnyNTHeMnllYIVRZyuCyIwjGUtBDg8XgeHF5EwT6tQHES4geCqavmKCjgg+SQ3nCUOU0D3ZiG6eT8nzT8jwCDmNC9YJnq7n9yGaGTlK0dBJU6jUr8l0V2sAPDmxietFkTiZVkW1ZF9aRHkcTf78SXvJfC7Mcu004bpzt6bJ6jPXmo+kisFpmgd1OrccpuvbbpnN+XLqVNX65xZj/ik3Qy1ZYUTOOZDv4boBu2HAbv+IrhvQ9T2w38sZAJhaIZpgTQeI1AS2CiPNP7OB1t5CW5hLVnsmkFOCRCJnRcj6GixkNIWN4iSSyEbqdN6LIO06xIA8wZ2T4061bqVo4CPWdqU/m0K8fo6+s3Obvixjz4yZCHq2hcJHeUNcRZegHgspWYjJPrv2X9xbSVtvKYSH9IkMgy2CAMlSyEpDKu3IWhPtvwffm0PDeXKAuEhzSQtAIDhP6PtiZTiweVFqgSQYAoSdC/AuQh1H2nZi26biB+nsuYANhlSWMyapEECxLNisyTMsVylRHwJHWU1h/tmlDJHT59v8SVapKO/gPdBEml5gyMoUlTXqmKmT0EBJWYu+D7UFL8xnh5iWjGy4cvCuA7odqNuhGwa43SN8N6DfDaD+AfAd2HtEu5EsOZEZkkJbvzidK1yUmtKeWWvNdTsrb0sTyCcG8SFA1kumcErMvoY+Wj6E4pcIIWIcU/+dpJUg5+C6HuQFCopgOTWNGfBe1jEkSkn3QPRdD3Y9ouvFQuiGfJqZ7zt4J/fF00lGyzk43zfPnACSRZJ5w/QwHFE6CRxDFgitrKuW9H6NQFoD32k9VujYvEr3pPU7mqusL2p+FzwUE8hFOxwJCfdX3tDI/zGqbyLVrlYGCnzkEjREDui66XXNEePICIVEe/+KznEe4Pzi2A58ksbOoe9KP4njpJ0Zysp9N6Ni+laYYDa40xhK5/I3MhBF8/Ok8vwmMhM1l7dPlGvBhzVUteWiYGjdns5LNvxZPE/zesy0u5pazcuClvUtu2K9keTF8b6D6wa4fodht4ffPcD1KQunl0ycbCZ2eaUE2zmynSAywu+e76lNtYKSIQ0L8ZBJ+6AWQbRWwlQgWMdyERI2vFzWjazPqXZNToSCMFR1CBce5LxHFoykkUOuqk//yn4Au7an/g/K5aS7Fb9y8n6sFq+7ms9BRvaaTZFhHeXWOlDncu1gvjetFgqLTg1SnWYBRknOKHCKeMnj6spi55DhHMvQXeEGWYtyTiJIpY7i6M4QlZu25+gP8I7hvZMQQEobUZI25lw6WDtZCkPnM4NwmJt/DiocCB0VYaWSsciEYoVkqGBRMzBhrdnUmk6iVpTItROE8H6jj8SU1m8lDxCAtoMYyzInv2soGFQ0a1VQ9LuWN3eXhcwK+6SjKYkA8oDz6PsBbtjBDzvs9w9wuz0oRRkxdRnztpZ1UQpQnLSMvFu5iZjdmRTCUKqdw8ycYzack0OA9EXp5jRNc9ESCG1BEQtkNxEClkEiwcwsGj7Pwzm7TjYDav0lHNVYABMlcArFTNuHyZq1TFq/A+LbAJD3RtSpLuy42ggpW6cdq/pZVthYofAtaP2OZtdmIjT5RPXFiZ7DitoTYTRlTnnz2vQGTXSFaKoMDKSDrV1C9sg5o8GnG7QOv0s4ZYd+KL4MBMUXGV4njxNoKtftSLJVUhJGVd6ZvTuiS+alTmSXrJksoFSxTO3Mo+XYXLfjKT3tMaLHdAfkOdriOH5PdL5fdZ8SrDSro61Pk4kYmyriNTwk9kfvywZNzi+FEKFORA+C+LN8/4Bu9xm7YQDtHuCGPfrdDtztJCoGQCBOGycjBpdEPzOWTqbTCCZyhEnLVV9wJQLKT1bebQV8LRyKNl+YnAMQNSyV5/OvFXEk34sPIaTIo6hwA4rVoGus8122DtkPYAoCM41ISILDsHtAjMBpDBj6hAj4Dkwkjn8ga/4T34ARCkkFnPTBMujp9TQTUz1932O32+XIIZvrqGUlaN1aJoQwy66qVsJutyu5r0z6jHtaDG9Kc3GuYbbTNeXTjWz52cTmiWY4XcbFFmCIk0sULwbMLwDgOG1CQQQFAsWkOY4BMQhGClbccBrZ5FViOyoWiREKR3dE78LkxctEsRK/+GCKtgFopJRcn48nuQhPcX79BpNhyiLfB9XoCTD3pwCAaygnl9EpNhbt9Bm2Bc4IAjYwESWhILFmMj+GYcD+8RF93wP9HtTv0HU9gnOIWnkyAsh7gMfS0bp1LPM3J5QDIW9eYzMeYPP+FzSvDaTzb8lCmLXR3AOz1pSZOVrWZIt1UOpTRMFCNtq1vNa4rDfuSu4jm/un6zqcTidwBKI3Gr3CUPJAAJgIhCkV6Ejun4+VKZoLWCdwq2wNP9dl7DirRWHrVGvBRjeRefa1Z2yfo6uFwrlFSMZoAOrFPZ/BNfy0ROaVTeoWeGpyNdcXQ4RjgCjCBfE3MEQoiLMrgEPIjNv7wrjVRyHQVEINzPmsRz+icwEuhyOK5iI+Co24slqPzx2WOavWhU5iyr4Z9shvR81ZbWMZAWoymVp4lnelH0pA5dJIU/W9Xbr1Luc15wWw8DTLZKZPakRzte7P/2v+kqpK8J5eq7+rOpnvEctRgMsufe7gaAAR0O326PcPokR0O1BybmLSV4U/gBJmKteYSykxWAVOESHg4A2MenFlWLi2HoyauS+MVb5uJXP2IbSfKYzUXDPvh/OjKX/OZc1n3Zls1bmJkqXrglwSrowYRVFjljXmvcPhENMYTvMd1aTCy9EU5onGH2JhX0r90uHQ76Wdsg+iWAfy/vIMMMpi4R1lLpR7CrQFFGGz5FOw/2ZB3nhN16iQ630KjV3350kHBaZlrWYzIsV67i5SWy7WzNEKDQKxLGxmMSWZgQAnz3UxZeaVew4pgkKUtlpb0mupb64DOQ2Plcmr+zAAnQzy2RGhH/rcPhUwLgkT3f2qjP+hH/HQjygb/ZIlkyIp7HUia/gqhOXyotIDW7r0jA4Ru6y5WptNVivHCIcgh6YDIITM6u3nqXBJCczExpq/IlcQSPtE41oqbVGGQ9yYMpXG0ZAIWbOOIeXdSRsljYZX/s/58xE+MyD4ndRBHpEe0PV9cig/oh96+OEBcfeII5AWh21DqRM6Z/TQd9vG1E9nIJZiQ59hyNU4jLGkY3apr0q6B8DCONafN9mQJz8Wjp4VmaSRRtngFRGAOKYkf5I+xkYfab6jGOXsZrBD3hzIMg6d73ACy/tnxsgRgVnWVMpwGpjTJliZ450nuAgwRYwU4RzLvgUCwngEA+j6zxJZ5LsMvemxmim+KY9clruRcXh5LhvHKGVbNRFWzHqWtr7ZdFYEBxAFDMM+W0PO+QJNew8iD6IAZsLxeAJzlM1yEKVD1o7LKEXgEcNeYKPdwz5DR33foxv6ZLXqu6OpsmQZabZYt4mG1ULhXhhWVkbOVT9db9MLl5plzFVrauUICIOX2vbYypdMaVko2v6pBM81WO3p+XlWtqXVEBH23RH77jTBQSVktuRCsVDW0JVXWWOnPp3M1aXrPYkglocZIWdUOMeF+Xfe7AKlkD/TJDWAgTZaY3AuhXl9nZRZz4U9x5b2165WsW4bEi3X86Qw4+9y7DqIAJ8ijcgjOtlJ6roe1PfoUj4ep9ohGlFRavXlDppNjxausemmk9ZHVPe7RaVAnVlzEkZazd1J3TT9TqCcaqJooKV9M2uiQZt8W6qRE4GiMFxnD9OZapRZYXIkUUjq/NZ17VKEkTqZOdUJ8+4tBKztrdNQtJtKcvZFIufmZywDmCSzs1q8XcP6V/hRUuyMIldbCHO/xvzExfrVLPGkS/QuUmdvoa0dtAvRvrC5A2y6qGb1XLi2ZjGcTqcZ89d/62eP3YijP4nWQgJleJeOaaRaKEAcnomJOqqFgpi4RSgAJ1XmqVh02TIgwENPwyL0XVmaPjERWcsGs0VZ4L7mjxoRpoqNPgT6TIZqTDPMfKIQqGZl6kYRaWVczY8E6I5ja7Gq1adbFiWySA5hIXKSIsF5kOvg/JCOcywJ2txkB2tlvUx7kJ8JnJ8zS5DHJarnT2tuazl7T32t9X3qRNbRTtpptjZUcyWrXli7CJPcOBWzBGTOMgDv0kE4ydKuRijP96hWHxVBpla6zEmJ6Gvh8BNFjU0216pN1vFbj0k99tYfUNc/gcOMUKjHPQsXN4+Q0jI1n1o3X35AobBlIWyNstGqa0uhVVfLabNkJUjd7ayQS3WccwrVi/hwNBCNbsFHhHcx+zq0DUSErisOb4mcmkda6LXeO+zSfgw7Qb0ru8m9anGAbBYEsrGraUxAyCHAOhKeGA/+OGmbPMfkhnHSr9I/7bk3mqtct85LoiQSqneAapHqd+eSRpduzsVSZ5gZYsRLz0bs4CgJg24H6jo414G6R1mk3QDfPySLooM6nSdya/L+IwoQPVewWwznLRZ5HfpZ11kLhdbzbeQMIMdhcj1w6sAjSqiQOOaZopyfTOqo92BwkuWd5BejAGjG4qxcaHmC970IZYUhk7/EMn1ANg4KA3U5bbVG7EysHyMM9F/bxxgjjsdjPu1My039Cw0rU9vNnC01/d1aWrXzuOu6nOOofh95rXY0yadkLZGWcmvff00EG+K9jt6FULgn6WYj1QjK9fbi26L9c8Jdz5VdMuPXPLPWOJgZiBHBlNMydqLo/g6rfdiJ7QlIkbcTYdElZi8hhzEpYoTdYCYnyo5z541ZnLQyTxF7d1zUjvLizpDOtB+5D+qTN+0mAB27md6zpPU6KhsgW2Na1QI8lDI5+qPrQD6lMfAdSJmK8zMmv0iM2UbsufZZ0h2cb2ej+gVhALTTLCwlXavbN5uzRDW+Oqlb/WpZc86bxISNCy4vmQDkKEtCh7KZVISCF7+FKlCuwEmSMTWA0xzUXcSHwyHNL5OxVO1csz7tHNRrMcZ85kFL824LT7GWLLOvz2rW+i3ztxFFrd3JE2shka5rbZvNrpoG6OxckfdSLK819MMJhes0qfkEv2bxtehSCN+5ZzUlu04A9Wfo/1nM86ifCy6WfCOSLoIAs1/DCofy3RFwSGoaUXFMe4dkIZRoKwDoj95YECjOcYeUP4ZymhFPEQONZYKTg/MM52JZlDnE12VUoVgMiq+S0SSRP+9MqEHxOSQ/SB5f1Q7LTnnSEdVbrMAiSmcdDIDvJZ2CGyRxnUsH4PgO5Lzk5LdRRvoi7IHd2U+VdvuiKA92HtTa3lYruDzuvHWwZBVYWhJSMniuwq8ZE6uOnGj/LiUGdB7EgPMdohoVYMkEI8mLBKljgJLGL7uhpS7d7CrrADlCyf7ZvEUhxDSXzE5lFIXF+klqRUsjEO3u4npc2mN3npm3xrVWklrvg8HmWNIISocRqdCVvuZJXIRx1ab0QJn3VM4FWUM/nFB4T7S0hOvj/Gb3XVj8Yk5PMVnO4oFmTEa+q0WE2dzQA84xqcPu0JTqZId4QoftZH4ds9XQVdq+czzRRh0YPk6tE+ckZ1Wy9hPTN/VkU788t8ZSVVA9OCMUZpqd2V2OAnOJJVMYhTXVd7sdvPPofI+++wzXdeCuA3WSt4iSLwHOSTrmrktpKyg7iVlhojTEVJojPhPmaR4X/Xml9XiJlpSONcLg4rPJ5WOSoWm98xBHMYHoBJCXKeg6UOdAFOBYtFSKAT56RHYZJmKMIjgS1BlTBBM5n/wYSSP2lHZ9++y30MOKfMqFFCLDQ4IFrAYuoehtoagQTgghn3pWM3AL0epnsbJKfZrPSGGiehzremtns22X8owxBIAoCwdSrcjAwu336zLTmFjYG+fADykUNk1yckmzbmvltba2hSzOpxr+OYf1Wge2Xm/9cnGzCpn0IHoptcdGO7RJt/lPYYxoQh4p91V2taq2ZeGlMJ7mk9cmIzSMH8YlSYZv1to0gIZQmPfR0mQBgrIvpu972Xy232O/36dwvw4P+704lJ2XMMAuOZT1xDTnUgrnbTgtiRSUEV6hEKwpd+7e1thpnUvO0zO15vkwvYfKDv28cYvzztyYUlXn+RNljoyjWJapODKjc7KzWZmr/qnQUIsAED+T1/ml1q7xjWXtmVpwi4zDOI75z1pqtbXVgnik7HKZ2Qg2hI2GmFp/IzMjhpJK3K4rFQq10jR5NpcyP7xQsExoTdlNQiEpyS3TTuub3dLQvBavEa1q/9LvS/eKMqbaZw1NtOqiwnRTBWSZP3NOnyBwU0jmJXIfippL5v6KiSjnJoDNJNR00JEJzvWIdlJKYflsdqlTBtrzBQW8DMcv405ATvxB+hMBzdlgLB/nOKdf73sPDDs4t0fsHhH7T+BhAA87+P4hRRY5uH6AS4fiqHnDqe+c3o2mTbfhBpO20HrGbhfwOStzSz32XwATxt5idEv1CSbNmGa0TTAPQbR4AkAM53tEZjiKiH0KFIgBHANIBTMILhIcx6zN+5S6HkiMnONEgE3WiZh8ud2tXEZLfEK1dwsdtdb2VHGZj+/StVqo1+1Q4WV9C9oG/dM8XRJ2a+sv7y7GaKAybaMsCPuORcgWhWwNfXehsKTV3PoZ55i/LVNHBWi5loCoP9trS47DWggsWxYMqhKMAYXptLBP3bhk+XeeLsYPEc2XtiZUwTBkMkDqwkutoewU0IymBLi+Ckms3i8n/0E9dJPvNuGV6XNcv0itdaCRHzvs0PlP6PoH8O4J2H0GhgE0DPApykiEQpfO9ZVQVu0PQ3HutLkukbXOyqcqnTrQZED2u86Pa9IXMHNmFOfGaM2aK/NNMevU3zwB07nmBIkKSFOKfNqMJ9wekQQ+oig7/3n08PAiFGKE06ynaR4xMxCCbHyLMTP/5T6V7MdWKIBK5E+rX7UvYWkezQSSKdviA2ssDTsf1Rqw+6Zi2jhLJDCafZ7WV/Y0eFhMQe6p1/Y23vrdhcK3pEuC4VbkFqCCSyGp9SRbSi5Rk50kE5NxqX1WcuCyQLaCyzrt7Pdctwn5W9v2hV9W9b8VMZIFcoLDvPfY7/fY7XZ4fHzE09MT9vs9Pn36hM+fP+ekYx3KZiGbH0cyVCwrAvbpVtjZdAdjCE0GsW4s1pMymTpk2dbfGrMWZSZUWmgYDiYMO2ulee5HaMhuDBIg0XWd1BXlcwwjXEzz18v+BI2MUm04h6HaOctTJkg0TYsfY8zBEEtrUAXvErxUX6vHbSu6ofdbeEzDU9W/kZUBX5zjNty1bgOgSsptFeq7CYUtE7wlhevf6oOut9IafB+Ymov15Ki/B6t+L7T/Ut36udY0mIszOJUyj+KZlp0nhoFkKgAmwRnCtXIGWlOirrH0afpLmZjJQrBwE83xnAI9Gd9BKlQgMlRCwI5HLUxpUq7WjEhx6IQ997sBu/0eu/0O+8cH7B8esE+CwGrWrpM0F0wF46X0vTyW8xd9ndoXIgg8pu+K1PHHAMIMDqmpDlA4p9Hb+dISNHWI5NmIFzMXZ3t5TB0En6wmFgc6CdPmFJkVY0zJADVySOYHkYNTmIS8JLmLknASCoOkMxF8jNnxK5FMXfJdyF6GCNkLkU86oxI+bA+6URilvbbmAsEKNlvGWhstK0Cv1XsVlDRctVb61Lelzzkej/lZerwoAHASmiKArS9Bs7raOUL5t7KRb6ogrKG7pblYW/6S1nIObtlKayS7fdlLkFEua19eo/wlTbDl4GpNrPyd5p+XSLHIlk673qJUZm7D3qyQLJMRQNrEROY28xAqm9Qm4xS53dfyDTCCoxxRWvo5aTEJjKYMwnuPYbfDsN9ht9/L52GYbQ4CkKExQA6AydFQrjDx0nRuXAMo5WuSaxZmm8ex132u0y2csyxq4bH0mzI2YL6bV8u2mGdbUSPz/5A66SRNNUseonyEVIxATF4XUpR8lIgkz6AuwkUCcRQLwhlLwfxrz1sWzcOBE2SUhbmbbtA0jWxb4MwTX4Ltp9ab8f3KEp5YStV9rXOXrbCohbxutgsh5Psl7U5RPPRP/DGS0UDryP9REQAqBKbRf9sO53n38FH9UrfTfYTZ96QtpusbnpI/nUuJPKOlZqmRgjYWu6llRJMFZjF4SgxDI4uGYcBut8N+v89RR84s+iWo4FpaktfnIDV7XKVtj7ZvKUhCy62FglRAXBJKa0i115alUgs1ywQn4ZiuAxHyZkAduYlmPeW+5dnVv7Nx4PY8qy2EmZV5xjpbGgfbXytklSxk1Pd9jnrS+9XSKX4Fs68ChalPTo5M7bMwpyTek990Z/c1yMr64zg3LN63wjtvFwS2MZhouT8KyWYwoXokRPtKTEOvXYCxplq7TKCzlIPT6/vLv3kBARONJVdRv8McEbv8budWg30mQDEkOMdgzQA0OV/nCcPQYbfrsRs6DH2H3ssObUTJ7sksmT8l1xKX/RMu5GdJ1WSsqhQZpVjSQl85O2ZtOUik1QKTWprvl9ZRixm3xtFqvcw80YbrspbkurH8oGuJgQm8aTIFuC79zvk4W6d9TXPacbqTJb+Whv3KQIklABcAGiXrrjK3LFB8id/nqQWkUTli7RVhqwEkVhmohUDr89L41EqF3mc3y9UCR8deLRUVTjYKyTkHCmobFE1/+jcVDnZtFsvA5lha4iRtuoulsBXiaUEnbfN1vWZUKl9f9L1R7melMemnYD6fvb++vkpILmFMc+3MajUEWnx3eplsJ/Jv516UGYfkT5FkZwXvJQhGrUerStpjTumzHTg6wEli7+kRrIlpK+M3EJe9XoTPJSoZU6l+Z7SMcbfGbIu2eo6sUKiDIM4xvPQN+Z2nPxXUWUjY5zsHcAqScCkmS98Riy+CYoRnSdFOSEJKhQKoHMLFDLvJsjDImA5/L0nvdAxKv+TQIisUav/AJYvDMvNqhBbna20x1DSLkgJMvwQuK0jjfI1ZwbA0T2o4yfzSbHNN7wY+Wloglq4xhUC0kgleB2Xci4gowzaxcYoEo6R1uNTu+ZitiFgoKvKqcS8TUVuHdL9t29S8b35euqYCRU3l6tnquFOz+XQ64XQ8woFypFHXdej7PsNKCiktRerkPl9i0o32U/6hfMIK3N7CAku0ZZ5eEkD22rp6bQJDHaOWVhzBPA2s0HOMsyYdA5DgMRUKE4ZKaTczyjwXOE12TzOXeut3w+B0pkPZuVz31+5+tgzX9qM1lvqEPAqm3Qr3LeVC0nL1bum+7xO0xdBswa226S5uu4vanv1OqOapamAGVbhE70YoAOcn+zWQ1A9NLFkmAcCZKCTLbjMREDjmX6vsQK3KMY/oadSbqHLt5n9dXrI8PbzElJm2YC6Mmq+cl2RCeoLCR/lwIvkemeTM3zhiHKM4k/se3W6P3eMnPDw84vHpMx4eHjD0ckZC3w85tTjlQ1Uq7Xc2KomBy/Zm27L8f9F05RAaRvUODRNuYfLn4IutikutRV8SCDMYMnWG6iFBsokcAC6wSREODs510JMOQQQeRzg4dOnQHo4jOAZ457Owl4N7HOAkUskyvHGUfRH6yvM5Cupo9pLwPYJAyRKxIZ+tsShwy1Qo1H6c6fjQbCx0rJWsH8EKenUqq4NZx6zrOozjCHIj4CRgg4hkt7jzKQutQ0wHc6WjjfL/UwMAchlys+9pC4J+N6GwVZs5Zx20Pq+vfDJsPwxl0zx/q36clZ+a+dPf6vsTDNN85gLVzIKsFTC1CM6x02UrxTDBDMDY65XFoeaxmwpMFR6SqmKXHM4P2O8fSvRRP6QDTKaHmNSRTTgz93IfeD5HmcsGt6aEa/W+AQEA11uvS/h4q86LliaQjR5bVT3jakGmJ94xACYI02b1IThwAJimkUREwtSZBEaCFTTOwTHPHMXTcFsqKCOX0NOlPrdg0HoM5+O0HPlV09JzbJ+tf4OIijWsc1IlspbX//P03emczNAbJYHf6Ms5uoujeWv5c07SSy9qRe3YJCbfETWFGdG8OzW/hPbaLNaZpJhHtCzFuNStyGzaaEEFr6Zy7qa5l7TtMG+EpI2T+iv/SXFEKJRj2tuIpNGUyvv9Hk9PT/j06RM+PT7i09MT9rsdHvZ7DP0wSV887dxUsFrmr+2ddqL+zPNrSO+yGsir4NANtGR1LJ0bctZSrz5Pplyqv4ZNdAiIZE4oZJN/dwTEeTprPXOBmXNuIobYzbYOKxBqnN76EvRvGIbmeNdQzpKfRf9d8hm0rLtWmgzb5lpgTcNJ53tLrJCZvrOp8qT9twLnXBSbpbsJhWvp3gvlRyFJKdCYnFRxm6Teh5ZgzfyrTCT5DswS/MNmtpk9dfI4YLo4tG7dgRrhZ+Y1Uw0kaYVuwmHq3cMCh3L+3KlsID1MyObbdzkE9fHxEZ8/PeHx4QHDbkDvO/jqWcB0ERZhRo3xadhfua722nBcqomN/RiT2ha00ksW9JJF0RII15C4zVNadk5CUafWJWuGIM5m836UWen8ZK/wjQeBEGJERwHkQr5Hmds4RsQGI7XHVgKyq5u4ONcvbXy1jHrNeMncn16zgmXpEBzbhloQjuOYLYiO9ZztZSumRDjJcrGnzi09dy29K58CMJ/s58qsIcn2ux2HfQ80zWlSCYKaWjxsia+JPdl6YLsdPNcsa20pC4n0nKbJ2hzXWnpMLQU23zPLJsP0THus5h9jxGk84XgUi4A1/K+OlTcLTrHv7NSu2ma/0+S7weTt+GgZQoW3lzne+lxr7UsO6UsMYElw1O1YR1YZmWql9XNLkTYDnmjkrN8J6nnR9tmkb8yMEMv9tca+xPCXQn6tQtMSBpN5MXsXU5+CHedzgr+GkupQWu2L90Wjb73veVuoWae14CYb+y7QbYRCtoyvxT/zp/R9/nJrs2l927Yz+G1W0fqyRCqf7D08vZ5sc+aShdPZPlf9Pwe1TZ4NZOE4ZbGlPURz40TxScsTWoJbPpewVFNtKaffnY0I08ytqs2ZZ1dIkja7LFYLYZUFoadpHZy4wruUTkAFR+v828lDW7gQmQ6Y8VvyV4kwK32IMWTdpLVw7cK2AqQ1F+u0Ba21cY6ZTNq4ktYoadP6UkivKaOfNQVJTD6X4s9x8OaAb4WJxFpg+MAIKcx10j9TZy28aoHQ6se56DPxi3BevLJMeQIH1lYzjJJB6XeNhpLUHOZZVOpIozZh3ud4oW1nrejU/buLUNBMmS2SU7/mGT3TrxDGVzMQG7LVIXNDhGwStTRUWfxbZNnttP6ZNuHmkPe5p1omJ59LPiPnU5/1qCqDtU+goYWF3CvO2mJSPGVnrm5v9fvkWcwmxTVSJlY9HGcKDQC6SCsclVXaJCHHziyGND8aj6fMiKO5oGevCTOI7OAiwYMwBnHSda5DP+zx+aef8fnpMz59+oTHTw/Y7Qb0w5CSr1GKo7dSKD2mEtqliLFg6pE2siO/tyDpj/W0LDXx2xuP2sLW/mYX/5IPYKnOVr115FHzuQ0BZgZhRnUo7eQe5yVTagrEj+wQ0U92zDMkzTmck0PDWfMgRQy9HGwUXQCPIzpnUkgwQCy7GkgSBoG7DoGRo3FOY0DfSx4m8gzyneRccnK2gTL22bg6gbY4xpxkj1NbwZISxaEcNsXMgCP4tD7GcUQMAcfTUXbUO4ducMDhNc1DAnmHmE5Zs36Z2iEt6TkKG+g6QtfNBcPET0JdWwFaoKt8Cm2TallrWle3akZXVbFMhOnCv9iOhWpaA9rSZO2Dq7qnAsH+a8e21H0NtfrKDaZb2wnNbiwwLNXMp99NmRZssATF6Ps5+4p0UVAOUSzWDudU4JrvSPci7Hb7nAG1N7hzZsiWyXPtOGy83BXTSC2r8je1Auo/S60jIVtlLeNaYujnhIG9b421UNd1yc/Rusb2vVF5e8RV/SBEUiE6125rbXg2pqmMvIeqf0Tl2SveRyar+dVWQaOP5bb5uQ6ac8laq/ZeBqe9ClKvbrqzdUrR6TtZUi7WCgJLVzuaS+MsySKdNIQbjKLqlBRYww3TQt7KMG9nLMwrrsflQlk2n8sNVr88W8mF1uR0ZPM6KuRjquWehwbIMp608CYTThcFMEdZ9L5FRuLmN0xLlN9JNMVcX8Nk997njWk7m/yuFgpmWDLYcGEczpWpGXXtQ6ghCjt+UyFS6miVtQzpnLWwRGsdkK3+rmEwtn31vfpbUR4wm0t1v9Uqqi2lVt11XiBUY4nqWUvCYMliqAXpDDbCfHzrNs72VjQgnVZ9l97TUpmW4nmJ1uMwCtnMBkuXVd0gNfJdKWbIaodu0gyeF7b3YFsH7ycQAJu6dkKc/3f+t8ZnYYEypjUMs0TCJK22XQvY6fcJbN7uwZR04lohYLSuqbyRU7VQXZPyaUHYfk18PkvvXucHSXprlDkgi8uh7zs8PT3i06dPeHp6wufPn/H0JNFHu90Ow65Pm9RcQrLWz6EtzLCV6VR/r5md9sP+6c7bOkyythRajutraEnLnVl/F56xxJxsP/WzjQZyNI/E0d9aGrlu/LL7D5wzqbTTM+VIy/n+hFqwbBH2tu329yVBWPfHOYfT6ZQjh1QwaD6ksqN5Wrdte2nv8s5ruVd4AtG6c0ksXe1oLuNUBlXTLlinmpRYkHIw2gOsR//CBLwvp99I08Xaur78W+LO2bQuuVyyVbSxLaXmwvmp+t5Q5auv56GI5kKyZZrwUfn/UpmLZISStkMXVd/L3oSHhwc8PDzg8fGxHJ6TGIZz5Xy01kLawvgs1Zp7S8uvF6/eY4VIzRBn0VFVuTXtXvNbS+O17Wi1u67bCum6H3V9WVBAAl7rcE5bRy08a8xcr83G+QIEZ6/V+1XaqEh71/e5++qx9N7jeDxmwWLTu1vrqR7DuQ8q8Qei5YCJN9B6ocApd0jSsoiL/CEibSYAiZaJldDQcjAWgpqOnDVmM8Bbe7Jwz9ah2v5co303v18uaz/nd3vFACyZu+fK1lrOkjmt1NqMM/nMmGkmRDQ5utJ2bj6ZL7yxbKAkTcs5dN6j73oM/YChH7BLkFE+dMWch8BAnrvXCIaalvB9y/zs56V76ue3hElLeFzb7vqZS36MJU20bntLUbg0lyxza/2u42Y1ZisUWvN2qd1NwbHwzNbasH1eutayGFrvi1n2JeicsDmYQJIypiRVNAq3EQoEtXh8+S39ou4wVbqFP69Xw1YLBR9ep5i4QjnSDxBcjlBiIklwlSgPej7Pd0oOwQiG87QEMMSFieuKDFpV9zayranvPldbzQjQ6P925vRWbWFJm9LPrfL1ZyYgWgFAgOiEpqxD9n6ANhwnSOl9pj+fvnfeYTcM2O922O922A077PpBwhtJIuOYCTHBWnrU8FuYql3sVttvMZ+JdlzBRUvwRh11UkNGts231BK1vrV1tpSKpXpsf1r16L+tU+Ms5KJ5g3RMam3a3mfHv7YyWlFgdh2thamXBHVL8DsnBwhZodB1HbquS/Ugnbtc2pJEQIaT6z4VK8ODmODhs0Bw5Da9T2CDUHh8eDCdTPgpl8A8+ek8PLC0ENSYtHgagBynXxMtZZBsdvwam+N6WuN0alERDLd5tn5vt4FRjsZsb3+vY+Hrz5ZmZVrFzhgEqzHPStYQEfq+x263y5CRZj5l5kn5yTjwun5dsrTqv5YQqOtaEgotJmbLtwSC1aS3zDVbxxIUAmDGaJc06K3CqZQrlkKLKdt0FvpZfQmWuVsIJfsumOG49MHi91pXS+lZGkf7Duz71ntr4VQLhFpwhhBwPB5z24dhyEIhsrS/ZanUNInQYoL1NWzdn6C0Wij89OizpcCs2hEQInJn9N/AjBDLUmfJXSCJrhReMPCTSxwx/5Yoa5OM7IIF0SRuntOLcaiZKjU/nqf0jBvQknA4t3AmWobcuPZpi3DVhOE2YCudeC3Tt27rJdjiPFOoG3iddsvM+VB233n0Q49B//oOXefFoUzio9LDimrQ4lyb1wh2ZRK2zBrhsiQQaoFSM5ZLc2jpXZ0rt1RnzSyXIJVbWCi1pm6fYRmtPrPepFWPmd5Dpu5W+XP9PffOl+Cj1tjUZWsB0T59jYHoZvPD3tdqU+pJFiZLY72GVguF//6zYv+6IIRnnUYgBBUQaTdpiDgGPVpOEivHGBEisnUxZVaybJnKwRgA4FgHI21AAUBUdj2Ko6oV3VQ6L7DSFgBpy0Rvp5/Otb1B9aeF3EdLz2lFWpxtAxdN1AqG8vwpM6jN6vXCYPJQveNy+xpEADiOIN8LZLTrsd8P2O0G7PoOfefRe4feO3gHdM7Bk0uGd0FVyc0ZQ6s9S4tfP2cGVGn3LcbSYvA1452FVKJe9GYsTLnaqrB1XxrfFvTTclxeqq9lUVwiIsqnqdnn2bDNmpFqpM44jpNy9dhJXqyC18/2qJyZv2v6Ua+BpWuWbKrsev4USAxAoMn5D9KmoiPaNTu1IgsUa3+vlb5LtFoo/Pzzz7MBYAbGkI6WYyBGefAxBBzHIhRClAEJURKDKR6ojTydyiHaChkVwcMTocD41oDQj0GbBEJVxp7I1SIbQniJtuqOmwQnUXYi63nLkgBvyGcyPzw8ZAippSmWqt6m5dpwyBYzb5Vvd+m8g3Sp/NL72sqYa9oKNdhnXkNWGALIY1rPy1ogW1jIjolNFOcb7986qWsI6C1KnLarHAc67aOSnTfjOOZzHna7Hfq+BwCEVIdN5rdEk99ZlNpakNZtuETrfQp9qK4kZh0l0ogZ6V9JN3A8Ta/F6GTLOTPkwCWG7rE6ncgw/yLdooGkokJQWmfepsUpA6U8R8vrbyN1k+h/HbxYCpVfKvjKdHXxerZgctCnCjv7YuQXImpKtNb7ikymbsHpK32mXCPZgl/DQ2zVC/MLcdYnJn9alCDCOz9PLbaURjX3gwhyVq/epw1qCajp9VqjbU3+GsZyzsGTpA/onMeu6/Fp/4CHR9mfoMKgpRXWddX/noMFWt9b2mddtmVltfrWutdCUzVDtDCJfq+ti0tU96U1Rq1+2HtaMNKajKT58xmLBCiWgoWGgHKmsWrXzTToRPAo/gYtb8/PUIa8tA9kKcW4Xtf7AeS9E/X7qAVbPdYxRozjmB3Nyidqy9yuH/k9Zn5nOg0CJcUbYHZgjvOxuUDrhUJ3mF9kaYii/wrljGPE2OmgSqIrYfhJ02cCc0ljfDwlyQ1GhI3mSMnBkKCiJChCZDNQKbdIgqbsphZmxisRAqYLf2JykbmOOWOw99UUqAfRNOcQgyUXFAtTjSZ/k2ucYVDGsb7kkyAU3kswzNdqN3oDOSO4pjt9tQ+FKEcy5PcX80c5/tYItlqLg8aFchZ3+VkypusgC617acLaReSc+AkcS56jjhz2ww5Pj59kw9rnz9jv93nh27pbFkLNBO1iv9RmoHLwNcrWf/XCXGqXlq/hA0t1O89p9+eE1aV7W+XtPa33tiSkbDrp/K8rhyTVWnv9rLoNenwlUKJ39N4Myzg/SX1iFQbvPU6nE4CpQ9vyB9tmJSssrLWhmv3pdMrfrSDTe23/LB86nU6ZB9lHtuelOJSZ9ZR2vYHLeoyC0jh24s29h1DYQl3Xoe+G9E2FgjB8ZcLqRwDE2y5WRwSTCg4RCkoxl2WMQScq5fsiy2eFppDq+H10GA0WN8XgppNeD+WwE7COOLD3Hdkj8HQx2/prWGYLxENYNvlamtu1pu/SfbUW2IIl3grBXGqD/pYXU9LorNZXb96pmdgtIaNztPQu1kBC9X3KaJb6cY93sNTGt0Iqb6Ua6lEFwe4ErtvuXDmtbCs8Z6m1zurvVoDoPFUGr8dvAm3ryyoAp9Mpt/vWQ24d9mtotVCgRSVK9XgCQwfAmYO9GJr51BHnK1Pt2JfrROVHA02oUECCj4DktGbO/7L5LkUZu5Ez405BT8lSQbIMCsLCTAhj2jiS60r3JMe6CCx5sadIGJPFU54dwVGFlZhvnDocgoxVfp6q2g0L5cTAyAb2yuVgrhXbIFJogTbVNbUeHIjn+0iyK9amnmCYncAisYmmm2nq32x02CUi5mS3JC3J4mTG2PEO6DxhP/TY9R2GvsNu16NPEUeyH4XznyYgKe7lZTq36NfQEvNYgmTWtGErbbm/JfBb2v1SndZBeg9qPddCiCoIrHCYRWwla/acz0fL1+/Pog0tZUjL6G96spvWNY5jFlgtoVBbChYWc84nniF/yl+17mI5GdMedd3CJygLGClTh/wv0frU2RdL2AfGyXcS/EPAoeY8apiWkwlaoCn9TZm/9UFIWdZhBADsYxEiOkAqCOxfqtk4t1Xrn37XZ8QInKJHiFPTs4avrKmZjxbM9Zj+Ve/rxIRjLGcGlHK2DUVSh4bUPmsFTH6bMomaiOZMYBGHJgLTwj6SVt2Q0NHkHKl/BAB4n/IbdR12Q4/9bsBup6GoHbrOwTuCIxThQCUUdQKzXaBFy41q/HZaviUQ7L/150vPtAypLtO6vlX7tXW1BMJSu+pyS228hs6Nff1sFQrKKO3ZC5wUjYhp/1rWwhK8toQOtKy5ei3avRS67lvCyfIL9U8oUy+KYuuvjEF7zIpgKOcprrdA7nTyGmOB+zdo/lLaEr181n9dxUjyACUIKsaIbvcA0DSJWv2SL5mHS5/HuENg35w4rYmiZuWlugHgxD1OXJxXdd0tgVP3YXGRsgNzm3EvMahLDMJczD6cNSSbbdqTWxf+YCKOnh4e8fDwgGG3w6fPT3h4eJjhxcoksv8DyQA9wzjPjteZe9p9Wi8QttR7SzoHp6wZi63C6K1k4Q/F6pVX2A1p2jaGhKXaFBKtKLTJPQvrS3+3n+s9KuesJw1F3e12E+Glzm6tz467ttcKF+3vFijItnHtu9qQ+6iR4jX9mTAVebhxZDJxRiOIPFpRNLrnoVyZG/2UBYCRlhXSYPtMAMgTOoSc+rlALuYFg2fyS9pb+lLKTp6GEE+IOAGwu7EVkjLtZPkUQjT3z60DSzFGBB5hoamJ8FFNItVRyYTU1yk8kOtmwljbfrne+cSJk8imqU1YjydQoqaWicrQ8JjhsboeEfwOw86j72Wzmnt4xH7/gGEY8LjfYeg8Ol/2IuifV/wyO72tY12fvW2xtJinXl9iCLVgWMPwW5DFLX0KSxrzknJ0zjq4J5TUeq5lispg6xTUGpwSGZOgg3NhybWFv2QhWLLjUzPsGu4CkM9GIJJ9CRqCWguszhdY25GfPlst4PQs5Z1NKx5IPHLGdM/SBkvhkmSyjNa2YJq61UbRAFMM7DIlRmt8E3pbc0GC4DiiBg+mg7zmmW3yPuToIk34dm7Rcx7ty2+Io/xJndO/UkdZyDH6xWfXWkWEw2llWm6pe74xrgUxyGfKu8wXyQYZVJaQaXWe3MNAcmJW5zDu99jtZH/C0PfovJzmZXcuO7MgpkOdmG3dnObztzHcc3BE6/dbWwNb6luyDup/W1a7LXNv6KjFwGvBYK/NPmPqg2jVuWSFX/qr21lr8va71fT1T4MlbJtEMOj8kB4ofG2fZfsSI4NjWyi0xnEN3Qc+4sqZmi+3JplqjdMJVg+E/Z5z7a1Aqe5piotGMp8glzTGNcTRIYblBbmGlp7J5DHS1NF8rn2bIDa0rY0puQlnnjOgsjAsPOC6HcbdA/oEJ+nRhi2GwFwn2qPmx1Yf/gx0DkJpKQHnaEmo3otqzbpmlNIWADTf62CZdksw1taCYv1rLAU7jy30VO/CLsrO0IiiKoiHVQKXBAMblOWSIFj7flYLheBO7R8m6AuZfxKINEV7ypccGwIEN3VMK5ns91DsGWBwYFMDsonUHAiqYYnztKRBSctrAcAL+vYckoKNrJlBJVXJyAACzsGGVO15sOdZ299Kd2yDAjq4MnhAnpBIjl/LUGXfxUJbG98m1kwzEqnE4Dtn/T12bugB2BK37RwDzuPZe3TOwzsvUFFe6PI+5N+lxWHgPzZtX5BhLcbxFoG/dlFe0sRb17coHa19Bkta8LlnLFkStyYr6Ovdya3d+CII7OfijF6y1Fpwkfbv0ntTYaKkbbIbDLWc3X9ioaSpECmbcMu61nXqJnNcN+7qeemYlE7ty1jtune13lJobUiaaep1o7jhHTD3ZlrazaraHicczTKcaV20AJux/riS5kGcyxOi9cw8gVq3kbm+UvM+9+zp/XYM54t4ygAAcJy8GfXjCBZZmZ9NIddiRAUSqpqySBKaXYfLJQGRNSCGcwxOueYVL9W2aoK8WmuE6ccUMdx+GhUw7+8abWwN02xh5/fUultCpWWd13QPIXBNP+17XrRyzRy5VFe99uq/1vuwY2EtLP1ud0tb4aDlYoyTVCztuTWHQuruFAtJ/xqCe6KsX6YNjub25ebm1QmznPoLpo1O4aNxynSIaPq8bE7JxebO4AWh0Gz7whixPGT2+TwtaFbWGjCmrL5qTs4w/d/sc6N9pf2EaDSO3ALbdjIsn6aCWdOHKNPMTNbkhbHX548/H9aXvzNgfVE85cwAUobcSq8pL4gyLuycQzSHY1BaBS7tii2aYDmzgybPbOyMZZl31/DfGqraSkvCw0Ib57Dr+r6ldqwVZKutGFsXWrN/ocCiRqf1mj5Nbm6vgkXIcEEpmPd7Dnkt+Q1K25beR4LbmBGSdVtDOmoRqMCwu6lzhNSkbe2xqq22YuHbdmlZO+zr5+j6fQqbNiStKVUgoA6+9VP1PWmfSStcqG5GTh+yoixVZuAlIm5YOI3H5WeO8XI5WEazhNlUkxNthrBE5ABKmWYdFVuOwFDvdsmPlG7Inykr8WzeA2dGTPA58myJD5S+jeFktC6XIazAXpg9OYRuj0gE8jvsPj3laJNIhKiCjBwilZTZIJocW808pn+VGUg7OFsoBN/1q8fwnjTHx+dC4JIwqOtbkwqjxsjTl0XtZPZUPeGObPqVNDc4fTHPlDLVGsr94slcy3VBNq2Sd/DUJWioy0w5MqPzXY79H4ZhkrJCqufZH3MofxhBLqaQ9xFZ42QGuQAPRke6aVUyMQdUZy0YpVAt2hgl5m9wQD900vcYJfOvPT0qw3vnlHsuZeyYWeG6zUDItH5H89aq7aSaXE4MoJZua+vny27MSXHblg207hl1LNVyLZOFvepp69vMG8eEJo+02rtJvQsuvVOVI31mqDXHhQHkMtasXmzxtD1podoJLhipS/iSS3lyPJwXYWHs5fxH9nvrqZMG1drxVAPbQreGVNYy/K2Wylp/yAQmSsJ1M9l6l2CnJe3Jls3TjjOXZFiLwFhYpkp9Xh19VJpkBcLUn1Dm5/TfmTWTqnSQiLtKzEzh2TzH25aLnm9P+hxj7ZynK8zcC7RaKGw1kc9pOZccaZfq3Wyub1i0mzaFxGA0zR+M2P7DAsktrX4LA6TvE8hqZq/On7O00CQEzywAXcyUhEESDM55kPdy1OvC36TJb4B2vgWdEyQ1E1srdJaY/rJvqT02k3qWLPMb0Gxanenn0m/15XPvW8Z1+n0uENrPbCkQthGtsNNmH6it/pKmkrlWCN+Q7rSj+X60Bf8EsEkgXNGY+9V9T4oOhIRjpi7kiBRgNmbEMFpaLoWmlqJl030Thy4lexsh36vRFAAQqQfgBCKgR5Dzcta3/wx4D/g+J8VTSGSNgFhDt9b2r6ElP8Ilh/atHNNtbfrttOgfkafc5BmW6migqd9h+sxzTHypbmYDeumQ+cTUE4xFpvxqAU9n1tU3pLsJhUvWwLUTefN9fF9t8fuzkuuIEodPvlZwVFjPaPozNSwZE1xFNMzedZnc02nOc9iQqvsMbKEQknOSKIycn0BFaxbbGvjlWlp7/5JjeK01c0koLJVpOWPXPiM7u98gYJf6V34HllYQ1RbnWSrAUc0flnww9X6E6umyDBbei2YTmIyveV4rknDL+G8Z8c1K8gq6u6VwKVJi68LcGhbHfE3w4TpaeVrmN6DaEF9Bk4lEc7+Endypeja/FWmCDLAWJMksirqdzJi2dwofcbY0jC/BF6HACW5aYx3cUyBcSzmC7ky67yVaoxDVQmcJw166pu0pPgVcZW0vwXn2uzEqV9O1imS5f57KYi60FAk9f16z1l1HPdn73oMFupV+OPhoK8WNk2jbS7ynVNg6mbYIhkoLyy6BCGLBN1mZPqbVkr0hlvDPSVlN+VE3Kae3IOR06dznOhg7ICW9jjQA1IGoA7kd2HuQ70CgyYld18JFf0RaEgJbx2dmNdygXUvK4dvrPV9GYVHdNMY8PyTHWg6X2t7yQdxy/t1PhV1PdxEKduCmeN6y03lr/RtK389SuFO9259gte7tlN+VjlU67Eg18sYNYkmkzzYGi6ziz+ZzthKm7WTFpNQET8+b+gvKPCI3T22wpm+5FbdcwBvmYc1AbD7+tXDDJcfwGstga6TSNbj/Ofho9tsNVqf1E7QsEvU/CeOfCoNzTuGtsN4SPH6O19lyk9+/o55zl+ij1iRXeosw0Ps3tcX8/9YkPG9r3Rve9sqqDay/umI22Jdq/xb04Zmab0pP0B+qeP10t3AGpRjQE/gmtbEGvxIkxblYH+pLUL+C+jvs6F3yI9QL/hKksYWunb/n1sZbn/UW62mp/Dncf0t9Lctlqe5b8oZz0OJS1NHF50jFk/nI1q+h36t1IuvKti39bPQl/b66LXewku8Wktq69xYd2OpToA0+BYHDMzC+ovwVY7Ky6cQbRFk1/9bdYjfS5WmcLYA82ev3TrormzIcpBhsdkT7IoqLp8Ke4WAshehFJHgPdjuoH8FRD0cdvPMiLFRgXOoX230SG0bkG0NQay2Yc/6GW8IwS9dvoclvDg55I62BqlrWwiUfVPGxmTDdlvWbEVUCSDbaAUDUI5XTAmPztUZ0vyf94X0KwBV6zloNDtt5yRZA6F7g0Zre1ZtvplSbzelqy7+wSA14A8gpK2KGidwkr1Gryhqu3Kz5LdR7ia56zgXtdC1stOQzuOR4P4fvtxnoVnVjXufyOJUUDVsiE6eRP+uWa2nT+Yy/llpJ7ZS89+KbONfWyhLK9VD5LiXaFs4auoew/aZC4b1Fg7x1QNdOyFJ+I/R1p7KYRPwsC5WmhcfInc4mMpJDj5PVkM8SMuoP22emjLekn9OfWgTOg1yX/jxAHVJuDtnNbNp0DiqxkSF1P6YRKVyauGIc3wodrbl/TdmW366ma7X0wrC2qCfn2zir30yPlrDiScjbtmct+S9jnB6ZW/8r983zCdVtVz/Flve0hVoCvNWGe9D63Ecbj3+7FMplaWvn3srMW/dfA3FtQJryM+6hAVy16NkcSqH+hQQBCRl4yek0IWAsTIIVICICvGj4iADnE+bsLmkHjTgi53N+F4YHg8DOAW4AkiAg/wDnuwQr7dPv01OtrFa8xPBbaaJr4gQx6mHo5+CHelPUWmrNr3NaeisHVy3IrnG0r7FQzjlQ19Al302xKIX5al/te2VmY4rP4cDSf4JLWdaVuS+R5kPS4zH1KExmztcAwHc0ERR2jk3PPmif0VyTtttGOFmFxc7flsWwxDOL8/y21sKfAj5aQ2twyD8rKbMkApwrU6ac2zCd0GwXCBU/Qp78Bq7pui45lAnRd3C+E9M8Jb3zziNoDv0Niom0b4PvCW2l9Fqz/nvR0jy+3rp5c5PeDS35EWoltu5zPabXKLGt5ywJy1rgn4P87kF/OqFwSVPaykhu9fy3lr2iJeVf9QlMNxXYhhiLKOaSk4na2PggzmiXoAAPhocnB3YmysjvQb6Tz24HuE6sCNcBzktZ58EpJfalMb8mwkgW3fweq+HpdashrqV7O1lrK+ScJdy6XsNsU6b040uFNuPnyW91+Va/ZzAqDA84N0xanb4nrc98dqRAHSEizgT6t3TW30Uo3BPvAt5uzl4qs77t93tJ95wABM0wVF+3u5qnAiKzCFbnmpZIOrbmqXY2NYIDcydrwXVg6sHOg6mDbEwjoHsEuvSZBoCcQEV+QExJ8UBdrtPRdpgv9+IKrd8yDebpsY5b0qzfk5YghzVzyAqEJUx+7Vy8ds3Xzy4/oLnE3gLBWrin9idM72tBVWa88m9S9FyLBAqbHjimaWaAkr7eqR+lcV5MCy6sYbdb0Z/OUvig21HeXQwCkUBATECA7hg1ZYng9EAR3wPUCTzU9yCfdje7LkNS0RyWs/j8N/4+LYu0Z6JdT800vpXWtoVq4VWTMpHWb7WlAGx3Um/xmbXonBV2zgrS60vafy0I7F8tFIhS9AhN77VjurWHtZJsn3UuQKDVx29Bf3ihUJ/sdX7Scpb8a2ha843p3vlzaa7lTrtuznl2JWU1qM8WQmDR8EUodFkonNQQdh4gOZwcfof/q71325Ib57EGN0gpMtNV/XX3v3rWXMxczPs/3Kzpv6tsZ4RIzAUIEoQohZQHO+1KeIUzQgeKpEgcNkAQ8QE0TwjhARREKNAkkUXi6o3VImjPC+IIN519D+p7GWOiTjPdwnA9lPRR6Z7j8x6T7c69YzM9dKXH5Pny8D3n/B7+749bYWCdzPad1uCJVdi1s2RO8GcVJvo8a8lpuZ2gJhQn/BrW+xH02wsFwvHJK31+Epo6X6UPUPqRNgoXZgCs+xoQCfRT6pUhO5UREQJdBFIhIJXICNAECheEeQZND6DpUSyDcEEITYgMa0NlpTNZDGFbCLyeQXMtfyQQFC7yWuMZ+lFCZI957NVhCLfQcWfz2fZtXV77/B4uMyD/7izj178aaWQZs2f2upHPsEw5eKpeIwvIWzdKYYqin+wIhCHc9kb02wuFM/QjnTm/FLFMEmZGhizLZAAZmoJCcPYYI6YYMV2mElU0IcRHXC4X0PQADsVqwAQulsiSM7LZ1pSZi0OZGteoDl79+fYT4ehbt0zmjGB4T4Fg6/HW45ebrDxw7cvgo/eYd5aZ6ielhGVZsCxLlxRPyVpG5O7t/Ek4t9J7BG1Za8U+O4SAOTRF5GdYo7+kUDjTUSrVR/f4Y6cd2CeuPVv2S+bIYYvIuZRNCeaaWK9hvoBQQkLDY907OcYviFFCSsP8iBiiLOmfL2XbzAiEGYgBuUQTLWgmOjMjU9n4XBtdHXdUhU2oQoGAsK8h7fXBa5zTH9WPoHSGsZ5qS7cz3vvQZpDHCx3NXov2FoJ+RuUJ898uS76berEKkHv10WSTDM5ZPimjpfIuFwcGX9r7+RnBDO8mFF4TIfC2JKrOCBLYggmOF/1xmcQ+EUZ5hNho57muNAZAF1CcEGJEmL6UtQQTaPqzCgVMD2UtQQTmS2HgEn6qEFSujCuBi3UgVkKZBGZr01hsESCAcnEE7+wY2rVu5/2OqF3XjwErEEY7eZ0Zu++p8fnImNfWoSvjBw3xcd2O+gLvl9k0/twJxbFi2B+z4yGEsNpHxfilh9QJYZZIpCoY3Nhif/3Jdr4F/ZKWwif9WHp8fMQ0XzDPM+anf8c8XzDPF9D0b+UKQipsHERYyoK1DGApgzvlVL/LdpzFfE9ZvjN3+11TidUOAQ2q4fdjrqMp6PFoad7HdC6/F/T5Eayjo0J4j5k2R3NzOHv4bwu3t78VxjzTKyO/wVZ9vVD44PDROTPmzFg6veXfqcvXb3CIB+pLOLFJOXOom/jQqsHmKRa3LP+xMjiyWrss8pJEWwV+YRY8nyQXfKBUywrMGqjgnm8w5ppOooR5Mhf4qISBgsBF/eYwA0FWFHNZSBZjwON//V+YL2IJYHpEKgnrUryAMyMzY0llQyPOiHlpbc2l/jLaO43JM1ztA1nzJl6HDAbRs8BJHDClhzoxY4xl0qi/of0Nwcbqr3EI7oIKjEbKDXJELvdXTfDY4rluq1J1ynNrp2VGHjd+icW6BTF4q4pLdU45yfO5PKkynlYHZYzqftx1X24d++79F+AydtUkxBjqexs5adWvJf0hFilRBvOClK9Y0jMYCURihaaUQESYYqxzgICO1QXXtXxbShvXsKJn8PoOY4z1mCbYCyF0x+v1lSdIEzjl1Vgww6pYLVx4ysabon6vknv0QSwFPlxhACfB/JPX4ww2b7/3KP2eBc6w7IiqI5URQDSDKEN2HIvF8ZpBZBhcgV5ImQERiFP3jLrxjVkIw3oMQVYIA+DyPYRQVhfPmOcZmL8gxBkxRsxf/oU4yQ5pOcxFiItFkIMkGUtFo0cx0YEyYEeTpvoOmnVQpnXl38wsz2EGI9eFPzkzQhXcKhBGjNVDQe4luIFB9Sh3F6+uuINxH4l2GzlkN3H1F1CNkJEK7V8zOL8ax2cUPHdta0v72wRfSYy4qgStDunYL0PCnlnXv4D+shBMxo/ARkVY6CMzr8qxYaBrRQ9lvO5bFHtk360KBe9H8SJ+6NewHbQnEKQADDpuk36+T+G9KvADyJq1yufMyfYS/AtRramUgRDEmRomUJhlspBg+hK2mWVTHGbkdAVzQspZhoU+omQF6x5DxaGsjNbVHUQIJBlJp3nGdPmCOD/i4fER8eFPxOmCOE3Ioa0oFi2sLFJrvL1BO97MH2hUHiqyUR9qQa1CBd+ZRpNqhDfv+SlG0SIfAX55Db02Akbf4Yix9dba8fL2aDUnmVfvbavMYTtVMXHXvCQA4ZiPq9vd5G49PdS0Nf7OOKw/hKVwBqM7Hwt9vPDzZUcQAhLnwtBEE2nwQadz9TBSKAyVIjj8gXmeEaYn0Pyvssr3USJ9iIq5K+Wn61csyxXL8zOu3/9f5OWKlBLm8AyQyd4IBiNgwRMCCAgzaPpSVhE/YfryL8QYEYoDeb7MiGGWSKKgaawDktaxcGxCGWjMCMjgAg/VdpVIIp2UQb9nrpNrE/etVgMDISBDV+A2a8q/q/cWGltx4qPrtupihd4/nRQybOP6fKecEbR25fIQw+fj4cVb95+p86+gJBwWCu81+QiQ1atn7jlTF35HxqHad28idMKgDgb0Jh/DYIghIkwz4vyAeHkqcf1fgPhUShRVnJllHcByBYULEp6RlgnL7Qbkmza2YYgIiPFSmP8F0+VfmKYJ8eELpi//jhBCCR0VfDNQhE5SNjKseC46Zp9zr/k3/MUsCGLxMyguPxIKW9hsIAAs+eq3WMdIMLxWq/Xle013dL0VCPbajhFZIPgHkMyrwXFbVxzQl056VUfWnT1m++yMJXgUw28CWADHUQjqqGyvxdd6wr3HE/QiAVLGyksCB15SxxF9CPjojBPkI5Gav1ykg0131WYcdwO13gvFnwNCnDBNM6bLA+aHRzw8PADxERwfSmm5TcwZyGnCPM9I8RnLcsHtdsXz1+9lkpVyg+Qj4vkBl/mCaX7E5fFfuDxcEB++ID79q/gzim1DVAVBTuoU5epI13a0KA6z+MdyF2ZzDkDWkFNUp7N2AFep2k88OU4IQ0hnPLbsMZtW4LX0UiHTtUcwsXcTCUPYACXc4I5P4W7zDl10n7wwEGshnip6i+lZgdP7MGg1Xu09ug35ZrlANz7t3zN0757u/CvG29azdHwcHcsfAj46svfuRyTGYnBR8QE0/UIok8IpQC7dzYjIEK09TI+gy38gPH1BuDwCj4/I8wU5RGTIJhqZW3TSNP2BMAPhkTD98b9wYRnw09//GzkvZZHOAolwiJgenvDw8ChCZ34qW11OeKa5OlVF80+NIRfxRqE4d5PLIkmoEwpA5x8AZ4BTD5WppQPUm/R/icopV3UWRBW3IBanH3W7t92n9wrTPEIv1S5/Z1pbCa/TvPdIAzh0BfOyLGvLAgVm5T7/kX4CrZWLI76nV1FRIF5CulLbW7DTNNXfR+inCwWFj35d0kEiDKvxSgJT8y/IRJBrAkWE6YLL5YI4PyE+fMHl8ogwX+reAgpDUZDFW/oUZaIhFNgJESECT3/8Ca7Jvm7lmQFhfsA0i6+AQcgMpJyxcOoGuMBBbSEZITf2m9fMjdQBhzbJKxN0WkuFj9Af1+9ey1GfiL92hGPsRfG8BR3xJeh1th+O3PtL0CGMaU1bzHNkMbymTF++uRp7loIvt/k5ikAoPq3A/XV79fAO4CM0gsU0FP3eM7bKsQJOtww9oyD9cvDRR5poincDBdbQ43Ky8PYWk515lkEXZoTLIy6Pj5jmR8SHR8yXB9AUwSZ2uXoGSMLUuOQJ0ucEs6H95fJYB5YKBYBAcQIFiULSwZKJ0QewavRUCd3LTShoW/RF1ZBLInFsl4nX2wF9H2F4prtoeHaIzdf+HjudXyMURpEl9u/W9aNrfvY43fIpAOcEZ5eD6g4dgUl65sRHiz4pEABV0nya7K2y9VMFQs5lfwMaXmefa9u11bf3xkpXrtP0t+ps7/XHXkPvtsnOyRtOlX3KjDyp6Zyqe42zBgIyUCJvyFoP8Q/ZbjJE5PhfkioiXkAP/4X4cEGYZlCcwdMElg2O7e7I5V8GISMhI4ORmIGUcUuzMXWf5HoCKGr0EYOxICeJ1QauNQyW3EYeBAA5AXUa6AIgoK47gKylQIVxNMtpuY64+BDudJv9UdY2ADACtn8HTfNp+y2/t4WgdA/HPnLte9M4lHKs5PfC9EVGwJuQ+r/em+45mr0lQUQtGinGOl+2GO4RgXCPtA5aduCwO763FKRpmsDM3foHFXJn6vYhoo/KAw5df7rzXw7R3Scyq1e5DJ6cwSihoaAW2RNnxCdJIEdxBl2eMM+zrC8gXV0sJIOy9blEcyr+z/V5goeK2Run/sVzWYmaUQYcJJ0EMyMQrXYwq/dWBl38AHtMjky7wae0ys7Qt8/YuX8IYe18P6c8rO/Zs1B0wo2YxUcQDNaC3bqed863co5LDg+fDct7JfP0z9upDNSJfKRc+x4VcuENfN9aFXt1sFCZr7cdW904c8z/3jMswx/BsKPn36MPAR9ViOIXI6uJkWGQZF5K/QTCNM8iFMIMTJJOAlRw/lTyAZFJ2GWhF+2oDmJXxl0eXmAYO8gS1IFsEs9BrRm5rw28ov3XScLVjNZJRrDjVtNKlLIGXIZaNc2zN8xfoib4uGlQNYbc9OcebTHoLYhnT+vfmmhHyre0JcDsve+leL2G3gLa3bK09P2+pN33YZoGSd57R1tQ5F75/n2Nrj9iQWxBokfqYuGut6SPAR9tSORfiZi5TqAQYhEEATkzFl5ko7OUQZRFS2cu+dSBW8pYlgUgRggs4aY5AbkMCpBANhwkFwtrxMRNBkWMyLfvNcYnl/owGIly5deyixk0tqjUW7qeSr2BIJEaJtXwZZoQFJoq/4JNn9HJrObcUsbuYQzNDumZroa/ekeZ/o3F6vIOwRGj3sJtj7zHEdwwun8v9NXXEWFdR/v3DCY9+v0RaA/aHTNmWR0fY1y9R30PlnyuIM9QLYPMmZGW1OUasovYiKju7mefbffgtkHEXhnRvEl7fXGvT/TZ1jopk7Eb91t9c89J/9Ix8tPho/emPW1w7/ixsgfl+WuKls854++//8LDwyPmCyFOmjaXsZSQTyqa/O12Q0kcpIWAIAM2p1QZdq3/ksChDVJWBkRA0gR6RcVvjEzbz/W7TJgycbKNJuobNtZ0e+1ahZKW2/fJqC/XzubO0rpjIWzBP0dpCzOuddvR6Eb1sNeItTNIfnaSfopgqErEffICritmgzn693YPgrLlHGn/0Ws8Dl/PmWrYOr1GO/dWsl3zwxD9TwWB+gm8RbBlIbzFGDksFM5MsD3pOCy7wAbvQdWY3JjUewP5WOl6T6iHKkjCBOhOZczI17+RQ0KmjDD9ASYgMyGngn+WhJFicTBqaiwuEzNG5CUhpYTr7VYxfAKAONXNb6D+BZZqVXuDSDT+LqKiIEPd4FfIp8BJaFEYJOZCtezqQNZ/enP52AleYSN33L+f8qD6NRiNydIRaOjeO7VM3Meqe2vGnhuV4b+fIWsl/AgoSbTS+9NOY/nPkm/LnjW1ZdUdFcT36mDL7c5DEc81w9XzI/5hr9kbX/fObX0SeGUh2NDSUdu2jr2E3gU+OjOoWW54R/iIS/FrQQAce7lbZAPWyOxdzLY8LuknQAhpAdEDkB9NH03g/AhAUlvnakovpgmFOeeIdAOW24KvX78iMUAlw+n88IAYYzHF5zqgYpbBHkJAYPk0kKm0glv6bPkNMEuaYkIAI4gbmQQGYTLrCCAL0HT9GtcOQIWPSgfXv1uaYBUaRkoREWKImEJEpLF25svqNK+d9+pNfD8urCZ3z2K5N96PCqcfJRCkfVibtoNrX1K+p3G/9XNyS8hvzV1fflcHd3yoJNAaduqYripojmlvPXevbqN6WBhIvzMzFs5DJQjoHcv3xsm7w0e/KjHfHfdvSvIiWCUDGBpqJth/SgkpP4OXv8U0pBkLIpZlQQKwFCY/qSMaAFgZNSMvwO264K+//sJ1ybhcLvjy5Qu+zHO9J8yyAM6bplv1tQO2hp8aLW5rbPWJxtQK2GbIfmJtafeqJHhMlcIad7cM25d9RCCMyE9W2x9bAsGfu/f835Fe2+YR4/X7Tx9ldGLM9mO72+GMucMxR5AUESSZ5OC9Kny7Wwej5aslao9ZSikJPMwMyvejjt6TPohQILzAQj1FRzSP0y9BtWvSkMyienHzD1QBAUbgVBTgBOb/QeYIhAsYCcgJnAMyRyRm5Dhjmi6AOnUJ4MxIKWO53XC7fkXKjDwxAj0gTiSfQHWjoLbHstQnMKDIdq5gKRtAqaxNYLkzkBrQ4/6yQkE22NGPrnEowFI93mqlx6j8DdIrrW+pOLarVqTF3TfX95jTPahnZM7rtTq5/XE7gZsg3U+i558/Ykq/Ih3V5EfXjoTCvX659z5HWrp5Yr1Gz3XRUDvvcMsaGv221oh//yuhxwyENtbUCe+fv9XereNnxtXHEQrvps+n1Yt4u4lJAEoObEoFU++xczU/AYDSteCYDOTvQBEKRM+YAHCOuN1m3K5XpPCIFJ/KDk3yAQBON+TbM9L1LyBGBAoI8YYpAjECITbFRjV/gux3P3NGZCoCwfQBqDJnYdBFuBHVV9NbE20CNaEgcBexCkcVCEbbrt6WXPtGPRJ11ba5lghlvUYpC+uIoD0L4Z5mOYICrFmv59VasdFGfsJvlTk6PhIae+PyPYjLQL2v2B+6qCt3D2Lpr23ntxitf4e+7/0ztpjuSODI0Kbhu6nvddD0VTmDevhx4a1ZOwZsZFRmmQtWkFiL2bbR1nVUl9HvI/RBhMI70o+y4A0+O9QijNaembGwOIwRAI4PRYtoL/3bt2+43b4DAB4eZDe0y+WCXLYRZGZ8+fIFT1++4PHhQZJhMSOkBIr9wAccdkqMxKk7vqov2btdUwvT1JBVYTBl74PNu/aJSCwinTDBhSnKFqUZsqNor+GNBNYRTc63aQQb1bodiPTYLBuQFYjm8j0F5EfBTmzG7Osu0ku3GfuYtjT47fLvacH1/eM+fCTQ0Losey3lXglS0iRzw1btQD9WONi6WDgqG6GwLVDfz7J8l+ijs/SS6IYzpdtuO6KdHW8rA5pFiDVdA7dT9fkEoj46h3LRgDmB8B2BCRNPAG6IUYTFwuJUvuUJfIvAchFsMiXQckVMC2JOiDmBlytyDmIqGEySITmSGIRMkjBDmqd74zYns1SNkGvWU9nruFo6wt3AnIGcwWkpaw7aCuwuthso+WOk8M42ofXADoA4sVlXXVO1umRCJsSNEesnmWcgXrscCY+hNrkBDdmy7L336N74u2fd/GjSkOrD1w/6fo/8O9qDa/YEwta9tgwLDaWUIHnJtp/PzFUo2M/RcNS9PrDj0ENL9rYj/XgGRjpCP91S+FGK/BE6L31z/33TrGwTizXLaTkMJIT8DGQgIiLQhMsEpFvAlYAlf0e6ErAEYJkQ4yRM+/YMvl3BzxN4nsGQkFQEgbPqgikCmEOJGApIvEByNsmGOlJdZZDq0LZMryx1q7KOi0BI4HQD103SS0ZXEkittl3NZNfH6z7SJ5VJYj4VzgGhj/Yfl3nPtPfXeejJjoMRJn2vnBVtKBu+vI8mEAC8SCBswXL2t6eXtHtXI5cLVlak9rEIiLU1MWjUSihsKRzrW49bq7UOuD+W7ymvrxUSP10o/DZ0QFNcFt3nQPZeEKJuGOgg/fLlCeFykaKTCJxlWZCSmJp///03/vp+xTxf8PT0hP/6v/+fkiI7gDIhaiTSFOpq4MqnN5tgoohajdp54zj2icZC0IlP2Bp7dTDTWPu0kNUQp8/rm96KkVbBM7ASjpR/JgX070xH/AJ7dN5aP16eh4HsM+waFfs37FibW3QPGhwpFbUu3ASDt0pHPoqfCh8dXdUIANXPeLhsWe7+HrQDjQ/qeEfj82WTLWQUutnMQWZNda3Hcz3ebksALwAIExL+iDPmPxekFJBTwLJk3BIhISI+PSGFB4Q4YQ4zbt++Id9uiCEiXBZgnkHTBMJFXLQxAJgQg+zzwNR2V2O2E6UMxiAVz0yIBgLS7TWJCFOcqsmjVgL5bqWy3sFEO4EIHIqjOWfTg1ZDD6CShTbWtCERzLkmYrVwgIVetiCF0YRU/0wv4NbpNEZl+HFiBUP3HG4L/Px9tt4+Vv4oaRBcXx1ua0ZM3df168DVypRkaJT3fbACio1r2ZbRxjh1x0UAyzmLz9t36BmiPTZKb9K9l5IW5jI94mm6IMWIv1JCggC+nBOYJHSbAhUVTd4Shak+K0JTw8teJuLbkkSTFQY1PaQbTsn7bHWWedUWo8o1RVGj0FRDBpDaGiUvuEZ9sQeJdq/pIF87IRSOL+s+4yOweuh70fFl+mdLzhjOrXHpgs+b30CPH6qwAMsebkQJ04VFKGSBkwICFhCmGJHwAArikM23G5AzciAElhBX5BkIVAQBijDQJ/X5h1bbFcIyDp0yTdrLoDd7P9TBz7UjuZYhE6gy1NBf06ws25nto7mWiIKk36DcTYQRnnxEm7NtVytB27G1gvqolrh6Fm2fP1O+Un9dWUzYy6FVnYZ9NIC0ur4B32UmnNcQDYDqPJW+7MtflhYV6MMybf6ho5Du0KfAhCnOuMQJDyHguzJ3cE1aWbhyl5pKFJCm5BDrDm3BCHhAFapNJYGNpUEwvjqhQFwEg451AMygvO1bsX3j+8X2wWssrU/46IOSTJ5e85ymCaALJo5gBhI9iJYRAm4FXsrM4OsV0zxjvlzwFGNZ1Db1g+ZeBXQsDzDXykzKX6/R+YU9W0x7pDG/ls4w1hEGvmUVHCln85xq3o681vdSXP1Irx3pWy8QysHdsphR82SNyvFFcBmnt9utWmk6tn3fW6Fin6s+gT1rASCgKA+aSDFGWQMUoH6uAJD6rjTHDHlNrZQm1w2dWoPnb1mY3ircKmOYdmNjfPi+H30/Q59C4YOS7P9czEhigBIiMRAn6LoBDhJtRER4TiSL2XLGlZuWlXOSvZs5IBYBUpamjZliGfykzmZdiwGGZE7KxTIP4JyNc7kfvB5mGWn2XuDYQT/cGKRoaX5qbDH1Yb86C8He67Ww0aRaQyBroTKyvLIR8F0mTqLVc7eeufV9zznZM+ft6+y19npJYdJr6/a8CIV2v7bfQ0n6XUOZU0q43W5YlgUxRjBzhexsP9r+7iN0xosI7Tnt6zhNkrZ+niVSrypGEzQCKRj9P9uyUOpEAoESb2vjXgj4ullmv/euRwLB0+hd2WftKSr36FMoGDqvqa0Q9DcmHbwSjgkCQtBFXQSOzczNISJmRsoZzALtyCTTokRjVb1ypBVWwKabFBbSEYiCismciYaTcdSPW0LCTyR/rDtHaBbKvZ4b1KNnZoP2m/YchYqOMlybWG7LMth65hnLx3/3zMEy7q329EKh7x9/Xj5rRtmX1VI82AAFFQ4agAGgY5peKPi+2HtH2tcCXcUaeBGp1TGXCDwySpBAnqjPoqJg1UWcGzjxSCDoZ+Sf2nundp5svafR39E1L6FPofBLUNNigKJNxLJdYIiyf8N8Qc4yiScI/h5iQJimsmCNJHy0aFvZQDxbw7MOYu6dlsrg7AQmsxBIV2gC28Jg81mD4/W+TQ1trxVSceZtYWCfdW/rwj1muvU7hOZQ3CvzHvPfshi2yttq5702rPpJNWgXtdNbXevn2OAFxc6thaaC4Xq91t+TjlfTTmWqOg66cbfBZAkoaBAhRsklNs9zyUhshUJJEJmM/0LMAnmWvjMCsltRP7TauPWVLu70Y/ueNWpJIbQ9zX9kqR8RPlv0KRReRWc6/LxVQVWryahrIug7GAsyRxBHgCfkHEAxYooBoIApSHoMCgSeAqYQyyTJQBUGvu6qyQCglg+J2EM21P2l1W8GI3WDcqSp6neFBFaO7k04xUaqEUbtQOfYtswLkOglnWBW0AIxXgbPG9OWJr5FwWwQs8WkjzL9o/UbCSst+57VVO/B+n5/n2yz3X5bi0KYorwzC/9UjL9YCzln3G43XC6Xzr9grYt5nru+2YJl5PkAMiEngChimi6YphkLMziXlCymnTkoK+QOPkLxmeScsfCyCkgYPd/61F6isdv+0z70yRm3/CmvEQZKn0Lh1XS080do+L3r9eUy2rpsLpFDAOcEcDGRY4GVYkSc5modZLPXM1ic1zxkpjrQ9bu5ohvYZP7XI01AMNYDctfM5963oG0eQj8lB1Jfg96J2zRKnZA0YHy9xnuPGY/M+S2G6+/bgoru0VsIBf99i6nvCzjvsRjBTOvn9uWjCIZGagGoYAD6kEuffFAVCHu/hZzsDmtSAQKy5udSRlpSp3T+MvnW57Uy/a8CL4t/xTrH9Xl2rwM95oXkCCYd0WsUg73xdkY4fQqFX4KMuVmOMGdJd1Fi6QJkckwxIsyzrG4mYNGQwqIYC3Ndaxj908ozSty6PK9PFwKijjmrua7lWW3UThIPJel1R5xrRwytLS3cal2j6+7BWluQiy3bktUcbXirMq8tIbJVh93vO9bKSHjZet/be0KGzb4wsRbb1rP8XszaL9M0lUWZTbu2u6DpX2aJWrL32/Pax/U5DHAq9THjMAQCZYGQBHUsa2v0nRApoFTaX8YNNx+IrYO1jKyQsmPfQ0X+/e8xf98HWwzf1+E19CkUPiy1Fc9kgqjlW9nbVqBPBAIe50lCT6cImgsGCyCnHheONFUwyg6fbuVmfYYQJ9k9DoEApmYZNOlRv1N3ZznmJosdxJ5G0EBPxy2uPWZon3fUHNf7VFv05PeWthku8+D60XPegkbt1N9bFsJeWZnz6vreyjI699DSGluPej0z1z4NIdRIJRUO+rzb7WbKbH0WY+wsBYWPdOyHnEEUQHFCSAFTAJaUi8SThZcZqGtJ5KHyYZIEdTywAHzmUi8YRutcrLAY9ZctQ/tAITSfehtAhd40gouZV0LV9vMR+hQKAzo+Qc/jheeI6qQz6jsAYeJUBl6MEfM0IcQIMpEewj5DEyrBllz+OsbnqWmJXLg9tepYo6N874CmwUAcWQqrVm/0v1ouCg0deU+e8a0Z2nYfjLQ0H8bqn+XLrExq4DPxz95ru6fhu3JCcO/cPmRk3hPWgqS7vr73vbnQR3V5aMU6nvuNm/oy/Ypx2wfWUpC/XD4Z4IzAUo9WI4gFQWoRqIzwgrLV3QopfaaHuvyWrqM+17aM1ir460bl2/5R60WFgd6rdT1kgTv6EFlSPxKd0dikS96nX6owsE5VY/KGEl0UTbidcGhGSku7v8bAN02IavnOu6DveMDM1drQaAzxcpDxdvQhl3qPFQxnBIA/ZxnulqUxOr/HKN3TDtXhWFn9ZK5wBtaJ1O4J5S2BcQQu2mLko/P+mb4PPUOq9SHIuNx4jzr2fN39jnlW2I7qNXrfVqgoqWBQoSBwT655s0wtoHMKGp3m+k8uK+8mEHxWhy0ox19jr/UC8QjZ53jBYtOz2D6w7++sJfppKXxYIoCN06yo53OcEaOE18X5ghgnhBDx7dv/lj0I4gSmXHLWEDA9SchqIAARYEm3McWWj0XHGWfDMEoVZNK1WmUsCCULKhsXs7MR5H83KL1w2GJKeo0y1v6ejJp2w5TnmUbOuWLVNk7cltVPtnXooK2jnYAWqrCMsu2RHbqFWAAQp9hpkrb+vq/uhcXu0YiBj6wEe9zCYb7f2a0rsNdVDdb5ntbU3rfWS/vOQ0U2/NTmotpirlo3v/CNICnhpc5ZFuHJDgqY6lqdEnSRAeaIzFr/pVk+XPZeL6/D52naUw484/djVP0TKwXC3G/v1efbtgPoxuOWInWGPoxQeOkkeA86Wpf3NJ5YOLZ52WX3rzjXl59TQs4MogUpAyln0JLAseDZQYSAhGEGEBWHLrkBXZkEYHEqDd0k6hkcV3FQtC3TXb5LvPZe24aeOXvafAd3XAojK2I0ufyz9rS+LY3bamn6nqwvwdIW7GSf61c6nyVfx63Mrb49o75qi816IXOmLubX3bp6Omexn9G8yfzfqqfj36hEfbUZ1YL243avDSOLa2Vt7ZRhx+DoupFAOWJR7tGHEQrvSWcn2Es68s3JVEFevDKdoinljAzJqMoAMsUyqAnEETTPBd5pm/8QZBEVA8i5OI/LwwTzbvhS1UZSQpyKtq5rF4AeMuCusncHuKXRpBhp9LViW91lJtmeENg6Z59lJ+09KMNr2PZzrww/kbdggqM0Yu6jcmx9RunCLZNlXu9E1xemf8bPOVrfPQXhKHWKh5S2ukagLBQotq7vL3XxzLRMFpgMyAUG9En81s/Zh5b8PSNlxp4bvU9rrW7136dQ+E1JF1wRSfJTpczXpt0ESYaXMwO8IIQHEM1gysWvMIlQQBlkuZmuzNwBQLIhXFtxGgO1HEcs6axJ9sWErv6sE4gZiOcn9taE6AY7YcB67pc7slb8+REz8hCMn9Q2YZsXCPa6ezCDvf810NHomVvCzDp3/XkrGNIAs17XTxjoS5WpGNfZdu+1b3Rs/Z4JRGUzKUDyhAEIrEnxUJa+MDDKAl2UKcCMb4+UnqTROFoL4rXAGL1L/TtyJuv1NmLqKP32QuEjwVJnSAc14AeFnaChjs+UNYZbnGfLskAyHSXJ8EItWiEzdwnaLLVrjmXfLDehMgVCV+8zZN/VCEfGYMDr7y0Ne0tz8pBNOVvTG/uJ6++zz9WP9yNYbdzXaUuIvJa2YArbHruHxIjR2HrnnLpwWttvQLEQuHFKvx5BytvWkvW7X5E8gtT22rw+NrYrifpxTTJdEEi2qiXIHiJqUYMH70lM5VWf3KM9pefMPffKsL9fsvnTby8UgNe9tJ9GHvSE1m+troiys4BqFtQM5iR5XkICQoMB5HquoYYwpenkTSmJA450n+TyXEZRraS8NXavf4/349a7GUE76/4QyEsn9prUT+IFhTIgPafMsNV/yzrw372G7xmr1QArOjcQCCNh8xryAsE7uIeRRO5ejeLxUWU91Iatzt+sl6eV8HfHz5TZ2qPAUPMiEAV1DXRnVJHRkU7d5NMdC71gOs8rRu05A5tZSw7olZK3pH+EUHg/Zt+0+bcmeeHKxO2g0YlevnOpBzFCmBCiRAhxTshIoHyRQc6EZbm2ahuLQzVBSajXYqgDScgrIRdZQBVT7buUWnlELfR1Z7BuMQJ/TQ8f6XP6SXHv9Xrm1yZ5lD6ERDTZSCx/nz1m62cdy3uwUYjrTeK3IKstKG2TdoaghYlGVsTIyrLn1FIYwVr99aY6JwXcWctgRL4tWeeHWgBMNc08ig1tlQkKJZqOG/Qkc0vgJ1TBCDXGu7b6cbElAEbvdk8w+HlyBI58Lf32QuEttK6fRSPssTtf+Tp3Wwh6kgU8IiRCCMW2bvhly+hY4oqcJqsCSFgyA0ZDadZL73h7bZ9vadGe9ibHEWFjSkJKx01thYs0BHXL8VitiI28PZ5eM9ltfLoNl9Xw3K1wWEtWeOScSyr2JlD9rmjy6cON14xrv012rHnY8B6NBFwdk8YQ6LTsUbnULIUKhxGhLXMmEOm4XrvVj2r6R977SKj21tn5aLAz9NsLhTP0oeAjoJtUXMZob8CK2iKDngtsRGDSkFEZ1JxFCIgHufxFH+ctkBEA6J7IQJs/DoYpKhbDLrtXhkMA4rAv9zShrePdZDjQZ5Z6JrEu30++0Xf9PdLuRjDQSGOszxm06T3I18eur/B94qEILxTsPbZ8Jea1Jbbuvy3cfyxsVv12gFaQXzdXinWgVmIZ0+1ridAjMi4xMgiTWgrrPthTgLauGVlcozK2+uwon3opP/tHCIWPxuzPkmh4XPl5HT6BClxTBi1nSEoXBsWGEjEACoCgJapFOYFQ8XkNe5VrWOEVhalQxZDALaUqmnmU1BLB/mQYYdn+Oq8h9VhvX84W+fN70T0h0MrR7CehvddCRyOrzqZBSJy7+/eE0ktpC56w1mAPo62TqPXQUUZOeRWC2QvAdX/tMTjbn1a7f61A8L8JbHKGKcyq9ZdjBEYgIHFTkkwhqDONud6jytkZ8grD1rv2785bZHqvf4feSt1TcI7QP0Io/KqkgylGk3URQF1soxeKKlZ8A4WZ6+DQiYeeIYHQ5bIHfBIvwyhyG4TVaiDVtbyWvQtxvym9lJFuMx51Wku5W9E5tgxPPuWCQkz5dh1aOiNG8BoaWS7eWrDX2QgurX+vdee7Ftpb6Vz3NO69c0fHAhHutIdNg7gKB66C4fU0YuytfmsFw8NxLxWeR+l47qMzkU2EoVY3LFf/HhxYp7tgA0J8E2K+N8JGNx26isy1DGjertZfVrsDwKlEH1EAISEwJAlY/o5AESEvCIvsqcw5IeUbaFkQGQhchEGxRCaGYKk5IKcbNKqJiiO2Ddpi8mtfAAAFZM6SK8YO3hBNG+wkQH1B3LVavqlAy1lVtB4UaNfvv2Rpg9QDJE7FbAaf3L2OyLGwlcBpZMoLrf56XFOZk6zt4EDihwlB32R5f7WF7n3K73RmwpGsHRFFQftIPoEiAjECRaSyD8eyLAhlJ7QplnUCqmhk7w9QRcD4GVKq/gsRMuOka3pehE/L3KmMTwWSXQmu9+nzfRI6S+357TeR+Ho4A3y7tXqqRcLtE/MVYEZkxo1m6L7luYR6ExgxXxV8Qg6agqO9Nzude4AJ4DLg6+5ttSRqE7oeRx0UVjGxfXrEyhzBnKPve3TCUjjBWRnHOfFJNUOKPsPl3xE6OqEVn9dqzaIif9Q/lAHmJBARAYGTLNBhBvCMkAOIAwInRESkdMNy+wrKGYQJFGZMYZLBnhmBGKAIRkZaFsR5LnyvROgUbTLCmK65YNUg5LwIwyyL3mRjk1ZdDQ3U1vXam+K3zpnHarqH4djinTdBhUHLHhM0tHAAArFd5a0WFXWPsxMzxMLEjGALdsvNIO1MYCCs4a/69O6wF4wHSPcXLhBfAU8gK9gZoWzZGigjcUZaMhIyYggITIhUhEJuQqEj6jVWzVekTFkFTFclY3mMmJlGuOkxu8bDbmSzaupAi/YQSwgBnGXs0jQhMyPlklKaSXJ3ZSCkm7xzIgReZHtOZiTM2mhM+XsFTTNZmGbjVQx+cFD4s3Qm7fGNppDYPthi7nsOZ99/o6ysI/qEj35D4pzrpiEar20nXMqpTSLaYqc2zHActw+USVqEIzMX/jRgEFvm9ztiTZ1GNdCs+knD8KtaFWbzU87juFv0s31ZFiLSHc78jmaWxgxm7Njc2lNiVIe9l/xW8MculMLNqrjnqBVLm4oCUspkXpX/HrDNURoJi/02navrP0Ao+Jj6t6P3HRaeYR0nRkYuGU8jJtGuOZf1C7JKFcwSzB2KH8J8mDOQZQEb8wKJ1S4x69VcFyaqsf3EOvhawj2wcNRqcpsc8g2a0bxK8v8ICqL610Qgsbtu42V02Dr6CeWv2/IZMK2vtX9HgkOf+VJ6kaNVETyxEZDQ488K0cQYZcW78zXop98tDF3f2jbV61uWit06vjUj3WLw+q4zBIZLrJltS86vnPRs+cjYhPZThQNlDBNJMIfasMB5RntuLPR97GkUGGCfseWjOlqH318oeKDvl6HzQoELxCL/lYVvOYNDBrGkvEg5yTHNgFrL74UCswoFiT5Rp1KNMoIIH8uQBBYyA7NYEcyF3VulXZlu+VgEe91+dBNy3EfH+seb5V1p5lhL49CMjE2hYa7R54zo/bTLpvhodJplXv7TsPvG2K2AsLg/mPt3atqoVqJt2xHmv4d739PkR/cPrwEAkoWZ1WHOZTyXneQk6bfuwNZ8I131qe0TokrF+9O2j2B1peuvLSF5Zuz9/kLhH0YMBgp8JEh7mfTl17KItVAjB3IGhzJx0BiA3FNgBgNHVdyWAjK4lGW0+XBk8L1P1MQe6artEfPai+ZQCMlTJyBw1pZ7J1Ie7bB2byU0CKkXBnYP5OoTQK7McCgU7tAWrOPfw8gnsTdG9p4vx5sgs8Iu5ww47bsKDnDLC1Ysg1zWMDAImXqf0/tQ1aK2r9ixkPboHRzNvypt2PZvQh+CFRQqGI6axZkhkUBXZM4IHMBMyOlWBo+uNgggvUc39albl6sJXVJ057bIR+Zd6HwKYJlUXFZXN0ukGWyNAbSNek514wkOLJqwuxe9EKjOTv3nJ476SHYEw3p96w8m1nUkTaNFWcyoCxUpABQkCGqKhBtYopHASEmcx0taSjMjQFGctchgAuZ57h9pBI9StS7M9xHmPTq2JwRsufbZu4IBhMxiuabMJfgBYF7AWJD5hoAMATAzOKf6FnNI0OWf4ASdC9mEa0ccwMxMfc8QdWO8WaNqDppXXQxtVdioO6GQbIUDP4WC0ntK9g8kFGQEy3dWWIcFBmLdLAXIeSl8PZjRpzBB2Ymq/tWBlYXvaIy7YvvZDnjdu2G9WnZU2f7ciXf0QjhQISutl4dSiCRMUb/bx1lhslE6joyFo8zhZVYUu+/9xyB1Ei5b5BwzI7HAinUbV3n1omGDZZGkq98IAhodL2dXePeWdbDF6Dvf0E4/VuHCCvcI5CmKkfylAiGJ6NDydMxr+LNCoOKLECvCjv9jkTwvoa3m2eOtP+Wd+n4pXYAQWm6wo6PqtxcK74kBUv3vZxGvfgo8KloPcQAHjRUXjTDnlllUS2AumVEpiEONSvhdYSLZx64XSyPnVBPlNbzVCAVTvXsRO+8JwRCRwFoDp2sVEMad7etVClmVy2dn20F6qVN2hC3vMVkiqlFEunWpvT/njExAoG2teA96MzVb1XPr2jVz2xM2G0/jZruNHOmUm9Ik16voLPARlSzBzGCkMs8JmVMLaz5Rn5c6pbecxLaP/B7N9t7RKvQj9NsLhfekFyqtb/h0M3mKTSkDvKh6lAG+gTWvETM4J8gG5LK3MzGBs/gCmAVyEuYpeEMIAciSJVUXAEEnXTLXojiZjaWgU5PMNX6dgY0oIoShYOjZ9ZFFXXKlbmNKoh4bgTDV42/i2/i5A2FFK42bGWW/1raBvWl7Skm2dl0kZDXdFmQi2d51CqDYrKmt/a71+9hKfPvOuQsfMZCZkDJJs1UQ8ALwDYQFMpYEOtVMwFQEQS5QDBWBQQQginUhdBw+ektSYa3ttokObV/YzXWmaTq1edOnUPiNqJsgFcwvsJBOIM4GIuei9WvqCk0dXYSKpqnWiV6vL+UqDMUM1tQayEMWoNjnCLPvBcMGrowz2H0TI/WZnQ9hsKftKyxKf+dLwgDfkryVwINjtv113wTDZJllK1YKVBZBYiUImoWwrYm+dUDBOYuhfqv3MrhCR+3Y+Bm2r7gqYWUknrQUzvSDff5WZNbIEhwJ5C0BvkfH01yc7IT3nAxny36vuiiW9x60LrdBFf0pHl21KouZq09g61lcnNUiIIqrTYUG2iBt2grATKgbmpRJt8XW26A0V7Bl9rRi/NQu6w70AnCslVpHcvFyF5N6nCZgc8q8grFtzYURJr93/iz5+HZbmhUKpBFm3Oc8AsSCCFNLxz2qr/y93z97foJ71+0xwu1rex9EU4p0rLf/yXz6chxzXkGNx9r5VjRy5Pvz9rsd/+9iKZzF9M7Qm5nxA3rJdnSHy+axVrxFZ9rJ1Rm283xah9a0/7lo89e+DlAeJ043pAxKMmi4QEYCJUUJQ83UYB0GFlsSR6QsUEMgSQ1Ak8ln02nk6vQGJIWrRh9xZbo8wGGauNi2FLjY9yG0VAmryVBhq8Ys7r0LFSaMMQP7iNQxzJTLYsUGN4QQME1t2ueQCqzYp8rW6x+mCQ8PD11eIk2iqFbXaFvUlVAqZaovx5Zj6wP0AseWaVdR28R+3lcl+ZmAZQG+ffsG8DPAVwAZUwZCWYzJLP6CGCMiBzAiQs5INFXVJsRYfQp1ESOPBYMVptrenDMul8vuRkxbbbfHtC98fijtS4WR9Ng0Td1+H7Z+9+gTPvpFidWrTO1IG0hjkHusUMu14k8o2lGxFABjXitq1OnuwaBULY1FrYeDFZqWYzQ/aqW9lIikXltm8sjqGuLfNIKv+iM2CuZ4/c7d81IBtKVl2+NdXfQwj2EIvX7UnxY60uv8fUfbu2WJbFkno9+2XcwaaVcs2pxB0I2kUrNy280iXJjAIRSfWz3ZG7bOqtyzBO9BPfb3Xnvv9aPvd79R0ei97tGHEQovmWSfBKNKn+yT2t+E6k2rwkAtDX2EhJp6ovpf8RmgWSKeacAJANoQXEer3UFRzfwZaGP1W6u300DvPLGr58/wEdwlNu+v0FbLthjTptZ6hzFtMS5/fM/PYrH70TFfhr12pH3b+3NJ4CjrDYCUM+LAAq+wkP41mhCXIVDrzVjVbatftuqlVoSS373vXt/ZMvx1be6t13ccoQ8hFF4CN73HtWcpQNMh/xxyOuzxG5mhUTxkd2lDBnHB3NkwhDJ2CwDRnkhUICJZABcCAUEc2ZKhWjlV21dZC2yQTDhX99rWBhOpab+eiHIdkQqHtcZ8lzpB+UGJIXsLl5+By9pFxxyUieRFIo1ySjUUdcRc9Pu9vhIYtTEg1VRtiCvQoCe9ZtWMDc3a10chE1/XWp+ckXLCbUlgTshpAS83ZFpwCQkcNO9WRJacqZJSnUPZpZCqXlQXcmagRr4xisN6XIc+rXhrm4XM/N4WdlvXGPudCy1sNhLq9rm2rD1BvEcfQii8J72vVcEYYYs/guxzDat7UVkV+mGqjuI6OVAU0fKQDO1TqtpTY/6qbfmypdy6+C0YYUpna92YlF2padNrtAkgDtTqaB6VdsdqaIbGelJ9HIu1MM/6q5EXCpzL6uVlqQLBMirLrO8xljE0eD+txZ6W7TXbLatitzeYkVNGSqV9y4KseyvMbc+FUGAlgU+1oqYcKaw8T6OWIPOE03ocuj6xzF2Frva13dzK9o2m49iyLLxQ0P62PrS9ndiO0G8vFP5RdPDdKxO3eR+5aMQWSQK3c5bliOAIheFGUy4aRMQM3RO6TqqKRzWsgzrcg+BY2k4b+nDTKga4lMq23EHRUIGAaklYOKyjD24oGMlVDwmfMQ1mfReSJVeEwdIYZ8Hb5T7ZklW2ZR1DQZX5Dfp0U8AegDJG12wJCvsse7xq4kUgpGVBXlLldkTBWLgwYxbdu2Zu41eEbvOv3YO0LPmEg3YbXHuPCOWIENr2u8wtMiyllsxPLd8QImJsEXWtT5pCd5b+AULhPWfzgNP8BKrz/2BTiRmB1LTXJfssOBFRkwoFq+8H1iT2ADFQmIigK7qmoZjsOYMQEGJJKoaWQZWybCAdQnBCwcJM48b4o2oF2CpqiiZtSIcXrzRUc/NHZvx7RJLDR5sVAiNnseCIMkJgcMrgvCCnG263Z1yvz1iWBd+/f8eyXA1EkTHFiMARE2lKjHsd0wbf+NqxtbGlxVqGO9Kca6nOCrJl5iwC4fu3b8i370C+4jL/gRhmyFo8HbiirWfWHdXWvgAVDl1LNgSAvU8tAR81ZffL1uO3YsmEUFLRW+vOQE++z6ZJBLgIhVgi8Cw056DbA/QPEArArzvbz9FLWsnuxjq52gGwdTJTYz5U81iYCcttty2xOlKDb3QSQ7SunLNg4YddM223s15LlDZoVInuJVHvIqpaoU0NLbuGURe5NOqhKl9sTT4MdNSoKu+DvrTM6Hq94na7dR9lNhavF6G99i90PgrOIMRqUWw5mFXr3SLP2O3f0TW+bRZeud1uWG4Cj3379g2BF8wBeHx8xDwvCKH4NahFz0WUVfkMxBBl9T4gW3MWEyJU5UHCV22f2Dr7+tiPbZf1OfQQ0ri9e/03stCaRdE/4x79dKFwFu/6iJPx/egYlHLs/N4jnCDQ4tifMTgLo+yho0xTNSqTkz+wbHsY1Jmsn6NSoL9GtX3AT4JmaquVsBpXTkiYZmBrSNUyyP025XwMoq4NXiBaTFqhIu9TUJ8EZ5YkqRWaa+X08IR91hpKsb6D0dgcRcfs0R6s5KEZzfi63G6yJWeQrTjnecY0EWJJ6RIJIOjWshEo63EiRRAzMhHAWfZa5pICr7SHqXeUb/kB9iwj33ZvFek13tHu/3pBPCr7DP10oXCWzjf2/RbG7a9t7Ol8vS2scuQp48k3Ls/e1UxjuYLb5YWCwkiAQE01MklNU8MMckZKt2IplPgsLimcSXP7Uwli2oPfRscZQFxNBIZAVNZSqO1TplgmnO7TK/DBgS01y39bC+c+AgnvHjMOtd4s47QWwvV6xbIsIC5Q0cR1Fz1vKQyp9I0XFGsn6rpuXTF3rIi94wqt2PZdn694/v4N4IwpygK8p6cnPMQbLrQUoSCwW8oZEy4IZe/wjBm5OOQTq0Yve5dT6fDlznAYKRC+j3rB2d5XSnl470gQjIIBtqC2o/TLCYVPen9aTULaYg12YK/v55zBSMKIDb6vGC0ZB/ExolVkhf5tH7lub0Iwt7p/HG3/5UQD4dq3q3fCqpWgq2DF/4MSPOD6zXXPSiAXRrnl9D1LI213K5rJR+3Ytt1uV1yvV1wuFzw9PeDLl0fM84xLBC4aqcOyLWdKCTfMJR8qIWNCylwh0LoALqc6ZnkA/fj6bdV5BIu9ZBy+JIXFETqe5uLNHuloT1ncuCFnPwG0IPv7/elktR3Wux4Y/mxzeh1r0FFtVhLQ9UxjdJW9HlztgfKhutetbN1ZJkKYEcIk5wMBHAGOoDCBIhDVSQ2WvPY51b2EGS0VBauWSoCa9VJ+qJov1ApAEwaWutQCJGGrRLJeVY6jfcztajdx/TamM4wvn2SSI0hgm5oVyOowtR9jyYmBxGAk+fACUAJqrqBU2q1+GCrFm/Dr+l1CXMUnxHVYyBtUjLEEOZf3BYaED3Njkgqb1NYYYR9j7NI31P40TNlCYbfbDd+/f8f1dkVGwuPlgjhH0BSBGMFzQC7DTBPj8VT2Mi/NmDkgZEYICcSy3iJRKruwybvMOemLMvuTS7khUGet9mkvSidB0rJ0/psyCAP1M5mKUqYCwH40nUUTDCZ0tpWA7VG8puOWwnsx2urEPP6AEVysY2oLw3wf8swVqBN0VYWmdVkcVn97hi487q3T87a+zrrLmt30oMOGgz9cBqdu4wkAz2WhKAG4Sl+ECVFhmRBkF7YQEKYHRCJcpiAZVQvDyOmGzJKymcMECYcM4CnWvR0Yc7VWoj6uaIcCHRF0oZpYI20Caapnrb8yFgm5bHvy6rDR3lAeV7vAaa2d2W/6aETqkLX3+++eVhCQq0dXPhiysRIARMimSiog+ugxAjBNAFECsAB0Qwhc52HmpTBswdhVm8kpA5FBTNWyEKFTGCsYkiFXBEkoDI6bZBJhEuQKFB9GCKH5NIwmbZmlwkO2zy2jfX5+roLh+/fv+Pvvv2VrUcq4PM6Il0l2TZsilvkiHWAoBMmGFEp5wfheiAJwuyHfgATxVXBOAJJ2vowratp6iGVdTrnErknQza4k9HQ8vwOFrg/0xYWAVU4jTYstlFf3NUFxHEL6hI/+gSToSRk4zEYwWA1jHcp2dFBVzS8zEGQRT4htlSuzqGohSPI9Uo0+tD0c2HLqAf5s4ZAtqMgvKhrhsKM2vcZJ95b02no0LLpf2NS+274JCFDNcxuO8Hi1Wg+j6Jft+vSwz9ZzbASPJxUmy7JUv8j1esVff/2Fv/76CzFGPD4+ViFj1wXsrYjW+imjtQvQ1FrJOSGn6+q+LZqmqdZB+3wEsTUlI3aQqJwDqAgFayW8B/z5QYTC2zbsd8GK34+spcJr00vJdaE1S6kE7Im1JOVQuYbz0hh+1f7KU3OxiaKMcqIJVLKbMolVgZJhVQVChZG46KQ7jjQvBEa/R4Jis6cG+K/e+6OFx5nn9daR2YvaQRBtb4mAQFEE9AZGbbX5FmHWWz6jflFYxL8jzcyqx7wgUAthBC0B6KKo1Gn+7ds3XK9XzPNc66kC4Xa7IaVUs7JuOXrvC4WApGEZFcrUCmodNb1KC/PVeWCfPX5nbe9za2EKNNVSYbR+2RsXxso8CCF9DKFATeO4R1sdac/bgfkpHNZEjAYDbUQl8SbTq1gSNLRU3p0uTiOkfEOIM4hsqt+y9y0Yur+4mtAUisYTJhECFEASF1kgq1CfrYKp16J6xrRnCdg9me09rWVvz+hfIjzuQkZWILaj8tsJP4XkLEOR77rYqQiFEEHx3I50IwHr55+FLuy5ylBNOTYPkz0nwQr9e1dhcLvd6vfv37/jer12+YdUsKhQ8DuVWSHghZ4+V+sXY0ROCTejydvy6gJNUBeVZa0jFRrbFm6zFAKvx7a1+GyK/e13pnDtzos09DGEwhvTS0Ox/imkrq7da6yjZus8MsiFdXqzF5CJLigsFyjJ5myJCGXiIQp8pACqMqvqdyIgL+vygZ4RjqIxtqCkl9JHskYrbm/IwiTMuQoDFQj6XSEIFRDyfkzYL/eWmeZHav1/vA9sf9m9G/Q30LR/7V/db0G/W0b//PyM5+fnLrw2pVQZud2qEpC9FeZ5BtAHIayFaD9erENXnx8Ddes+NNGftabEquiT4lnhYC0T30ejsTWymN7DWv3lhMJHmYi/OuWifRNyBYCs1UAIhtE4L6y17JjLIralQECNeXDKIBItLYcZHGaE6QKKD6DpEQgTOM7iYCYCKIqjkgRXFQcuVZdH3jW517tMVW1rQyCsrEwHTX0E38IWfNVTeT/+aGVABX4rcE8IE2KcEGPGNGnW0ogY5gL7BYkcC70/ZyQIFQW3/brXb1u+IP1tN4tRTV3LtUzYrrGwGUfnWdowz3MVejZr6/V6BRHV83asaBnWavB7HWs9iJsFMvJ9WEFj2zMSDP59uR5z3wMkjYzdv6QocKR/GqzEo2Lu0C8nFI7QR5jMH5kkRJC6I+0vu2NABfTrr417qTEnMb9FMMQoO62FOCNOF2C6gOIMhAimSSKUKhxFZeg3IUQKMzhG7SeUNa91MlraEwhd/9yBbvT+9xxn9yCjQY3qX8WyGxNCZRqofRTLpwhS6n/rSKjsxViAXT3IPNkLWaNU+D4VRtlbCnYBmhUKeo9aEcrc9bueV2avDF+/W+Z8u92G1oGvn7USbGoUQBb5wfglUkqd36B7K4NxcsbKXF+m4dcSbRZCEwaAWin6HNQosTP0SwqFkRn1SS+jHkryWgkNr9osSxlGVM1vAZBwuVwQLxfEyyMulws4XsBx2oVxNEc/mJHUjzd41SMI4AjjX9V9R5W658d6Tdk/iwRCsr6GCYEUPw89s3SCYB0xNF6ta3+rYNDPsiwl7FquUb8A0IdwTtNUBca3b9/q9+/fv3dhqhbieXx8xDRNmKYJX758qXX4+vUrbrdb5y9QS8QyfrUANscTEcBmS1O3N4LvHwu3Wf/DeBx5y2EfRhKB0FvB9rddA3KUV/58oUC6qeN5bNI7lT8FxHGqPUUoKzmBwJq1dPttMAyeWZzB4sMSuIdpQgoXEAWE6YI4/4H58Qnx8gXh8oQwTcjhAqapWhZcol+JenO3Ou3KpjyqYXr/AbB2MNfmbTDwj+QTuEdb47odD7CJCtWZz2kj3BOQtSMxIsa5RiFpfLySbmBkn+UtNWZGjDbKrBcG7b61UECJTlMnsfUZaPk2N5Pd+8FCOwoR6Tt9eHjo4vgBVF/DKI21hRq99TBUXJghGUm5rB3gkhZDrGOi1FlCtm9CLBFI3C9oDPpMQNaEQDX+wRglgDpFgwZzQhf8tTQdRL+KUADgNY3Dd72zCf8700h75fqfDsVm/pNxKbQzhamX78ySKIwpIk4T4nTBfHlAnC4I04wQNfy0hUJqHiXL0LpHA9Ct33yNtzSk3XZzH+1ij31EjX6LOuYLwOI4bAJNPDOv2jk5P8yd/vPMvt85bVso6F9mCxsV4Q4CF6FgmbRlpAoP2XPq7FZSbV/rZRd0WRzfMn/9rB3nZ6iH4uqsYWHA2uZ6tR1zcOxeNf4qnAr8M7R6eSwrNuaBhZOO0AcRCr8/rXFFwLO5/poT0MfJAc2rH23y0vpwHaBarbVwKH4KvUSx3csDLo8CGSHGFuZKbQBzHf066iszCwAANYdJREFUCQCNnlPIqMmLNYwzshi65hkh4AXCELcfCcuBdnyUan1Kg0ZCae95R47fo04wmOfbfhuA1ysaMf4jzxTm2363fEuiUFyvbSGYFQp2tfNIYGgfTmaFMhHVxWJtVXLzD1jBYNcwjNrnLQg9BuZVPaxDXO7ptyL1/TbyZ2z5Nfb61wdRtDbSkVc6pF9eKBzVED8SyaDJq2M/jrLkv1EmwQCIEEOsEEKxOItm2eyGVLKUggISPaJaC+EJgQLi5QF//Mf/EmffNCNOD0CcANKP5Diq+WwmLttzqiMUawbFVJnqRDJkLVPbGgM9fLEtUOrkNNFHWytpaw/mcWz4GG4o5aFfGLVFR8aCZyor2AeytDDnjNzxJ4IAtgFEESBGDJJbSpauUMHmJXRzWRbkEkoMoGL19Rk5I8zt+7IslTH3+z9TZdQ10ui2YDHwkP9YYQEAl8uliwYaMWXvsLak4asAuuslpXYf9qp9HE14bltzw+CSHE/PqwBr1zdM35a5J/C9wmCFWXet0ZTsPX7saX/78XGPfkmh8KsJgXv0oyEwWVZTuH454nuUEMoiM4aGNAoW3Va9IjyWgRiB+IeY7pcHXB6/yOTQhWkhgiGQkaU6WMuHgia+K+drXdp/duAPGfsdLdvjxfUcra89Qvc0/yPQ1FGrwV53aA7oNWTL0cWG8ql+g9xuIQprRmSY9VZfj343ptivLraa/AjSsXi/jSgbWSujMeCdv6O22Pu9ZWOvtRZD6ULXzbQSVtJ/sSsn54y6ctPd3z8P0AE5hDm5ZJoy/pD1eBiv5zlCx7OkvhPjoqIhHi3/rPl+uj4fROC8pI3H+xBdkKn+Vm0WhTGoeS+aSdGW4lQclBPo8lQWO80I058IMYrv4OFByoWkrJBntDBHRkvQxYy6AYLInn6Qd/VEr2WP0mjvfbe/R8LhNTSCqfauHU32UXlb9w3rXSEq/bG+n7ndq8xLEwtaoUtELsUCre8xdbBaeUv+Zje+aQxfBcGySII5Wz+v6asG7/P8WCGwtaeAh4XU0ax+BLuIz963RaM8TSNNXQRE20jKLmLL3ENSVsGRPpX3YMtcW1EZuWx5q23w9Wm0FkL36KdbCgxu4YdHrmfv5Hrj+rwRk/jRdEaIEIsOk3Nh0KROvwD5RkgUBe4BgOnfEILEftPTH+JEjhN4+kOghDghzF8kgiIQOGpKCwBs8hhBtz7X/YMJXLJyMlAnBBFqZAoRgUminAIB8867txPnZ9Ho2VvQ1ta58w+lpoGyZY4mhxAYiRmJ+wVaLaWE3DNNEyiKlZgzg2KEJnFTRjpqp+13uxCt7e62zmmUcuoYrYWXKrzoGLelIazDfZRSzrlGIl0ul8pkLSykDNmXrWWOwjrDitHKmJZoJAnxzbml2rArtnO2e2hT7V9UQWKVn/7ZUp5+APsarGWVMxXBBPxyQuGTXk8vYoKsOkSBb0jRBtVSAqgsZpoevyDGi1gHT1/K8YgUHmpaBARh3G2LTq5lOWO5fXNmOTNEYllvtuAZMjEcA9C/Xhh8JKGgFhK4B5D2NP7z9Vdt0AoEdN8Vcljpke7ZiofXuplyR7j1Frxh0z8IU+zhISuIRsJc4RhrBVjGPYK3rGWidbPCVwWLCgqvhds2jtrUNPxtYW6FlF7jLZZWHgAk5OwFUzBtGMNkAvGNoby1P+yc0/lTKPwDacx0xHlMJbEdhYgQJ1wuFzw8/YE4iwDAw5fKoG+YJY98COKhYFSsUxkhLBxEvT2okzJ3IafKkajCWu1/El+GY/4Vrx207R5eX8+/kxxh4cjDENF7fpDXPJPdM5Up17pUBqWM0zL8db1Glo0yQC/c1wIhV8vRWgu27BHDt+XbdtnnNdilh6ZsOfrbCgLLoL1zec+fwswIRAhxbbFaKMgLFh9G69vcw462Lvt9TkTF0m7+jmYsqkD/tBQ+6Q6RZbJoJmvmGbmsag0P/wI9/IHLH38gfvk/EeIFoIBMM5hl4c2NkwgFw+ozA0k1vCDQkSzFkWgXiTRiAFngDKe9aklEbT1DkP3ayr9+iHsmY5nNPYHQd8rbQ4brZ6zPn4WPLFMEChYNFIEOwGaRhY3tLwyauYYPZ91EB40Jiza9DXfVpmxANxY22no3I9+AlmMhKnutTTrnzwF9mgwlz4QVRtJ6WSd37w9YC6zeUiCQcSKPhIF9tlVY1K8BoIvOspRzY/4to22o70bmRXk22mI4RjWstWbmVR4XDJ9C4R9K1aELyA5nBShAcerNj0+4PH7B09MX8DQDFEUYZC7bEWZk9UdUk1YHpTFzVesrO3IJQ+tXrtY6mfuiCpUQEEPsJhywnqxe67LXDdv/hkJgT9OvDLFIM6/13ltf4anTEOt1BqazWm2NQHHaedWu1xqylL8WWCPYqP3uoQtl0FZr9wLBQ0XrOvQL3fw9HopSBuutAKulW2FqneEAVnssbMExaink3ATB3nu0faUL61Qg2fq13wXNNfeoT8U6yHsLSt8XVu2nTjAco58efQQYbPvItZ2Z9Q51eQeN8aV0pp0v6xMuO5zpOoEA0IRpfsTlQYTCfHnEjaJETYDFOaiTRn2b6Adfhz0bjaYOYqgm27Bve08IASGW/P6GUViBMtLgtJyfSaP3wPofrS0Zr/m/VR10Pil0ZDH9xnT7fgtltbnXnvW8Ld++L1t1i+s3Zr5Z05WWbXF//TvsUzMGNGuqXutzGlmHsz5rBD9ZOMk/pxNIUKcuJMC3vsd9Ya4MPqXcMXZVtrio/LmsFxEFK3TvSoXXWkkCmPu9ruW7grLH58WHsBS4xk4fuPYdBQLw85mK0plmnu2TDNHyAVmMlnAR/h7/A+HyJx7/9e+Y/+3/QJifcAsB33NAyrJPb+bmNAs6cCGWADIEiYiMTDJpNH0CQ0IlxMoAuO6R0N48xWYmT9PcBAJ0AgFp6Tdh8UzjJZDM70QaiLTWqtf9xGydshIxI7BdBtCSwsEx7p4IOSek1KCqEdTT1VG1f2oO0J5JNm3+drvV36PQUwA1jba2RfdMEAacuuu9oNA26T3WIvBWaFsUKMqRHosxlihKQiiOYS6M3ZYpQm/CNAEpZcxzLv19A1FogpSbEx4MLLeMELhj+g3mUqHcFCYLx8l+JLUqh+hDCIVP+rHEmZFY0/1KRlLNQX95esKff/4JvjyAQyxpihcsSQcc1wk8TzKRCARw2YCEuK245IwEtHBC1kgMqkKh+RDWu0rV39QchBZS8EJhzLSOdgqfNrPvCeOOCTv46EfTCIpZWwq981RSnlN1Svu6qyYqfMzH0q/hIg/JUGj+qLUVkztnNTNLhl1XBiBpMp6fnzsmb0NutXxrdSpZKMce04+FeYA+lNdbHPY6tQ6tYNF1CEQC0eoKbes3yTmDwygqCn35buxxFSRbY/IdLIX31M/PwEcATlZmqzNGTIAqtHHsIS+rs5Ruo16otZ/bVe9mFREhFa9UZtlmJ4RYVyNPl0fcSKyD25KwLG1xUwgt5bDs0AWACZypdCmDssmhxLnEZAcQizZTrurNW53M5tOskPJmmAHVeDk3ALYIIVKro5jyWE0S1XjN4U6rNuGjdkKa+9fvyT2nw4rkezXgiSqj0IZRWENHWz6RoT+BUPu09Se3Z9e2NKy6NpyK0zKEsigxtGoTyf7LRFDnZqiJDNvCLIE+eshoJBQUWrGMS/aOtth/AmAZaPN79MybkZLcy8y4Xp9xvT5XzF0317EMWaEpLzS8Zbkl3LQt2sdc5o8oVagbQdn+1tdQ+yQXhYnE16YWhvKcEIMI6jyoA5U9jwKZsWNHjeFZZD7leazz5wAdFwq0DsF6E2Ld/v3MLWeYpcasuIdulJxyMtfce07t+UP16L+PV+O2Ywt8fqRDT9nQRO3xTAEpzMhEuOWIhWfMD39g/uM/cPnjP4D5C56vwPer7HlL8YJpmnG5XDDPDd+PsZWZksyAkIH5uaQpZq65bVDWPQAAm129KAjzDwGI5XAMYoYTAOIM8FKZSUAC5ySMn4FIzalKzC1Gidf+pzqp69whoLxuLqU3pcC8i9p/MmFHfdrM/Wb3E5oBIjvZ9e+EwUgWptEynVZory8VLzxdfTRLYRLBCE7ROjUSSBKkafQNIQQGEMBhqpvSVLglFGERY4VBYpB8SN3q3yzhxBbP1y0x61ir+Hzv8J6mgDjFGtopW2r2q5tTWqQfSd7O7fbc+Sp0e8y//voffP/+vVoT0xSxLAHX69SVBwAPDw913wWbt0jrajVx62uwabgBIKWil1CfbN4LR9sH1QdTFNIwRUQCQIRpViGWcVu+9+99oDSEAFAwfgxuELwVfqrg6hw5Qp/w0U+m9/aR7D0354wQH/A4P+I///M/cfnyBXGa8PXrV3y9Em7JTiRlHOv9K9iYyVwY0ffv37GkjNsiESFxmjHNl7JdomwQv9d2a5br5uWKgTeNc5t+db/Cvb5ZORTLT8+M9jTfvWf68rcSD3oo7249zfcYA+I01UR1usmO19yVyX7//r2Guiq0yMx4fn6u+zoDImAknfW1swxsWVa4WbKQENB8Bnr9VoSTJgL05ehxq6BYK2qVXRUsQt6Qj6yz78NCbr4dvi+P0qdQeAXtY3hr2ooy2TyWTwgMwiCK38IlXH/LlRMYGTFIiuuHyyOIJDLi+fmKJQUwyubuRRg0CECZDtoTi7Yim40s+Pb1b1xvC65LEl9FwVHBjACU9QtZIWuBi4Cq5UODD4r2JEeyaOqizre6MAAWp92vSiNc2n8HsLpmBDWNBMEIi/aMw5bZ7mnXeOZmnzv62FBQ3wbLVK3fwF+j5SzLguv1WoWHZdQ+HFV8EbJxj7UIrACz7d2qJ7BO7aH32yigUf9bi8GHq2rbrAPbMvcQoozvWgeqFig54SyWLzeng76xksSya9FBAfEpFF5FltHeuXJHSxseL1r3ERIsXdhzD5U1nNvi94EiMmYEMKb4iMv8hHl+xI2B223B8/MVCTNokq0NBSZiMCdYnxxxz1CYGZwSlut3/M///v/w/fmK25Lxr3/9C/M0FU+CLJkKDAQIhh1CS9sdiBAqRqtmb4MgVPiAuSRvMv3KJfzpI1oJZpKPaCQQ2q09s17f3F/rGbMXFpbheo1Xn+MZmI8OsoJjFA2m38fOaXXYUmX4Khh8TiLVtp+fn+vndrvVFN5NCMjA1LJutwVEt/qsrfoC6Ji3F5TeEWy36hyRwF79am5bjo+wshaW9usU5/69Of9nW8ap76zM/np9u467u47Rp1D4ybQtKM6XcdRUVAVQIiDEV/D8/IxvS8Zzkkl5eXzAVPZUvqU8nDjIPTyQikD4+vUr/vrrLywpI8S55qvX1ZlKMkmKQxNm8oKRzcInS1vhqJVZvWtIxM8lD8foMf3rhYFPLjeyOvYYnL1O390I2rErg33EkG5q7yObLJNmbquNbdbPUds0TFXLGK2o1jbfbrfqL5mm5jvRNti+0XraMarXKWS11RbfF7fbrfar7j1hBYLSeK8DXlkmVojce1cjK/MslPQpFH4CjczvwVWvKhMYm7ZSslghYZpkYxwA6XbDckvIOYoPIWQEWoDlO5AykDM4Z2RSRZz6CAGmIhREm2OWzVseHh4wTxExBN3apQuOYIYIlxDAOZeoJDUCfBTLesHamtEVpxuhRXj9ZGL/P7OB+/QahqYnr7+HZTVr0EZCCaLWM0Xv8Fz3535gggohn3/IlrVeqNa/G4/Rj3rGpn7wQso6eNVpbhUMv+8y0PYk1nvtxzuVbeirvV7b7dsn9dP0IK29KSVZx5MziGQVfrWuQtPsa3kpIyWr1Mh/ot1vpwc5rvi9XDn6FAoflV7wTu1A8jlcusFUoClh0gByRloW5AVgjpgeJ0y0IDCBb1cgy/WcEkCNXTHbMkMRCldcr9eiJc14enzAPE2StgImXqswjJxySbPBdUFdqVI30XVC2LjxlZWgjE655MeQCSgsu/VbYeztCJnf2zCSXlmvqwVSH/o4sBpW5exYCd5h6q+xDNVbClvCuvYEW8eogN5We7fPZ+a6fsbv5Kbav44JZcytflgJBbtAzQpMXSBnhZPH+a2A1AWVzBKBlVLGYhZVan19OoqUEtIi/WUd4w2ik/7w1vDWu7F9eu/YJvQ4oE+h8JvSpkAw51JKuD4/I3EURx5P4DBLxMSSkIv2/7xYjNRGRqgZTQjUpyW+XC5l9WYJtTMMJDOAUPBml8NF62ctBY97jyytfY30JxOjE3jAecjP3ueFIHOLdbeQ0ciBqzSCQEbkIRr927D7W4VXRu0Z+Swq9AdUwb3FtObZrGynftGXhZO0zU1x4BJCPXcfKwys4qRWiH7UQS3Rcv3Ob0o+oZ1aMHqPh6nUWW7rasvUtRnX2/PKSrDvwtdjNL/vIQd79CkUXkFcnKHvQgTQa4pWpuN+gwiRSBy++YZ8+4q83JCZgTCBwoT8vCDTDRQLQ78+g5NqL+ppJpDd0AXFXGZgDhkIC4gYvHxDvnFJrZHBC8k2nSWrZ5gmgY5ikPULUB+y7ALHxZkdSBdThdoe1qALKnFLJnskAJCdWIN1NpYp/ShZ4if53m9/vaXOWQnUJIUj2GHr/pEg3XIMe8tDtV0reFRQjSJ8gEHa7PJObdmjVdXKDB/Kjn7WQlAntK5ZaKvi46osK4gt057nubMerBbvrZiUEggZCb0TWS0D65OwypeFoGz/e6czs8BKvs/XEJKu7wndMeVHNnSb6Bz09CkUPgB5TQqQ1xveCD+k1XfGRAzihLwwEovjDnECTRPSNSGEDAolo2NZLQqgbdEIArFNnhZBMQIUEGkGB8F18wLkCCRqTuBQ0mIDQMwTECOQ2tadAJBzqtpvtShCAKaLbTkUmZcPYyWk+b5nwQqH9/RTbzF8+/5HY2GrHGthVe3bMRHPTCy8uFWekvdJeaFgBYJd6GWfbzXi1XqG0BSLkVXoNWO/Nadq3goPAaiw0og52z61VoJaEFbQ6fM1uGJVhhMw6sS2fgjfX96a2upf+whJZrgdKGCF5pYAt+//CH0KhX8YNQbQftetM4M4k6/XK64JAImfoEIfRO07A8SEtKSi2VDdrxmzOgBFS3p+fkZKQJrleTFOEhabM8ISW9oMncQAkln9WrW/GDGFaTWZpGofxoHww4nBjqn0mLmSFwxH+mwE3dnyvVAA1nsZK1O1eywAQIhlg6ZRFFnF2Ruzs9tqhhDw/PwMALhcLmUls/gNHh8fC3wZV/VU0nNWY79er9VKsB/1WdQV3wwAJYiiwFI+u6tlyNZKsHUawjy0vaeD33vB9pHNAKvPHFkZR+iwUBg5q7bojFT6lYloH3a4p8Gvy6PVYLhX3kjzu0ssvgEi0a2D2g/pBs5BVhOUNBQCvQDqFCTnXA7MZa+2AMqEgIgQFlAiRBCmLNogpQl8LatCKSBB8sZQICw6MUNbYyBQUIn0mKOkyIgTbmQ0R83XYyI7NCtPnOcq7IjIbl1cj0nHTQWK4s13ec8Z23VtBwW07xTHexv7a0eQx2hiW0eo7nWsc9Ri/ZahWyzda/bWGeqfYe9RbVoXkmkyui0IyjJeD6NERASs00tYRm3j+m2kkafHx8d6jdb1drvV9ii85MtWUl+A1tk6qS0MRCS7rk3xUi2W0Xv1Atj2x6ivmgWCurLbW0/2faiwss/yCSVtHX66pWAH2XvQqbLfGSseaa1vVZ6lIwLieL8wgFyZoEI5zCy5hlCEnQa4KG6fLVgjJ1pYZQaYQDkj5OaAJo6lkAW6kTiXZWyrOgdZzCY+hBa9AczgGACekZalHY8s8BCxG3PtCdxq0syjrr9U+OV69V4/Huljz8jtvXuC3TNof95+76CaQejpSPu2TGaLua40VGaQETb6UYGgH/ss+7HHvIZMQRQB31d7Qti3RdvjVxirZQKgi1RS5qkhqLZ8FRyddWoc2npdCH159n2M6q/PVAE5aqeHm/x1o1XVvt+stTDq/6OK47vBR0cZZJ3GB/np1sTaLv98dIf5defq8YIc+3tc7sG67B1r+M+rnrH1HNSwThpcZCVG+V6OM2wYXahCxpbR9HlIOCppCo2S7RNUfRcAkLJs2kncTLMO0sgZTL1TklkDPrffwUezZr3mfu99NiHQflum4ZmWkt0RDVj7NKxmzBngyDUkVBLXSaJEay2oFWC1e/9dqTLYEEChh1GUWfs6+bmkDuGRtqzlpNTCZRX398LECxm7cE2/2wVrViO3fefJMmZN+T2KBvNt8e8lRl1Xsc7mautuaRQtJcd+EZ+C8pXDOu6bWCGE+6E9RxisZYil5JNC6+zTt46fz6e6nyeImCV9CgvAVKN0TLQ9VaFYrgEATsDyTUsBOJrvRttDG6x1cpZFQUTFsZyDlLs8gEMARQY4F+FgmB0zQCyro9kI6ne2Es/Q0TGxBx9astqkTRQ4Yjr6955Q9IyLmTHPD5BtFQi3262mmvj+/XsVEH/99VcVEFbDVlzfa71qNUzzhKnE9Fsm7K0c1eIV1gFQ1xZYp3eF08r3nFHrCABPT0+1Lhbjt8zf19My1qp0UOjatWUZeAf1yEKzays0ZxPMHEM3x3Q/EwAl2o65wLBFgIQgecpiDK7fcZh+ulD4SBP3PK1xui3z/7ywWIHgWtjw+DlhSYeksL+E7YkizclYCUDRXIqIkugkexOtvytOVQstey0wF+tAmJ7mQoIyN2NmE63tQamH6XNaC/AfSszdUN9i0Gfeo2f4yiRH0JAXFCMrViEXW8ZUNlICUIXC169f8e3btxrT//Xr1y4nkU1WpwzfPr9CMpyRTFinbUtnCTrrcGSJWH+FtEF8RNfrtQooa4WoAJQ2TvDQii9fj8VYcnQ5S8HDWyOLzwoZ287+PaJaB/7dWnjTp8ywgsinJDmrSP98ofDOxBXiMGSY2muIaO0oGpnrL7EeSB/wBmV5etHdPUp0sHzG8GK2q3nNxKeWa56gOf9Nv1LLbJSZgaI5ElCzR4JMfzGggupnywSlMTypDES+A9ZJ3d290gkA1D2z9yAKD5n4carnVbNWBvr48FTH3PV6xdevX/Hf//3fJaJMnvU///M/NX314+NjddBa524XdVSYV8oZoWjKfjtMJa/ZK/NTP4HWM2vUXLFYJMiA6ur6eZ67XEhWkAASwTRinHYPBUAtCxEMto4jKErLU2vAOorte+4FHzDFqbP8RpaT9xlo+Sq4ulHjeNM9+gcIhcEWPgyAtgGXj4Y1vzUxgKQDChZJszH9pg84WEXe0NhxlYdXWAFh/AVFGABl8JbnUJ4AEjgoRFnLQCEic8aylJQDKWGeL8DUUiNzznI/ESRJ93EH2/sRo2//WKoyry3P8bVWYPSOZstErEbcntHSNVhtWeGLr1+/Vh/Bly9f6rXX6xXfv3/H33//jf/+7/+uSd+0b1U7/eOPP7qVxN6C0XseHh7k/Zqd16zQsLmNANQ02FaDV0ZuBZmWlbPsGKjtUivHlm1DZP1xfYaPMJIV1gE59f4Sb3nZdsseJNr3EbpAbVm4wD9UYZ8QIig0a82XObJgbL9tnfu0FAzJvNmeWJ7OdN7IChlp8a/V7L118CZ+izpoLN5oy/T9YBzOZC8d9ZcKnPt92Wm0xKbcgVZMVDRAqwVnANFNArU8ABVLvv9+JLH5f9/MGlkH5qzR6u2xPeho5Euwx60wsWkeqiAxVoKFaITZCRN9enqqDOvPP/+svgSN67f3KPN9enqqsQp6r4U9vPYNrBfCAagOZLUEGnRFYG6ZT/WeEbSmbZfmrqObbJirWKZB/G0Drd2THNdACu3vtSBU5YW5WMHm/t7C6MfvHizo3/dPtxTOMC3C9kRYXXuWIbI+wR3cgRNa+fvPkVDIe2WMf98lcn1ifApsf5t68p36tqsbDNOlml4x4YN1XaFz1O7fqJIWnStWakyRnfcizKAklHM4tFZFN+XRrT4bNLNNbyJo98iNty2Y5G4xTjAwi+9Ftf4RVLElFPzaBisUVHsmCNN8fm6r2i0jnKYJ//Zv/1YhoS9fvnQYvU+FoUxchILAhTZCaCs5nhUEvv900drlcqnhsSFItlIANUGdFU62HO07FUx+PYf2Z/Nftb2lfR2thab3+WfZ/l8z+9LO0jwqq771Pev8b5CrHVYq0Mq+Iuhht6P0ISyFqtQduvZ910CYJ738zrdgLkSAburuFPiBgXKqttl2n5HI1hX80hZQr8hv1WD9tbS3aVYNS06LTAbKCQt/Lyunzcpmblokg8CKsSJgtBnbaAz9ipBhHWfca/57mqT+tcxINWVN1qYM/uHhAepn0D2Qp2nCly9fatI4/a1+BIV0rPVhI4P0nsvl0nxIpk7dHtAGMpmmqUYcWbhL2zDPM1JKeHx8xOPjI3LJXqrnHh4equDzIaVWKKjFoHXQdtp79L69MTPyMfgV4PbaGqlEJKnqAZlMjDap9LuWCUagUNZ6oO7ZLOph6hQGhVSP0M8XCu+onAltwBsbIalnhE4PvbyBdTAgMhOnHFlJ0Rc9xnJ/8/01QkF0/dMrQ1ZKQSArnkTzz1mOpaR4r4lQQdv6UWUSM4OCYX7O9H5zOO4AHXnGUQvCMnQdH34Bm2f+Hobx19t0Duootlqwrvq1cJEKD5ue2gonq31bB/E0TeIgRx9v76OQ9DkeRvGMVp3JDw8PBbbKuN3E4lFLQdtu92TwzlqvWavwsVFLEv483mjHvzcVNNZKsdbHuox+EZ1cv0YudI+GJtzQnRvV6SgdFgpniz7MGIqmehJsOv7ErYrsVNDG4u/XxNR7/WX9ewhlbVYCPWPsBYP/PTQfNoi68pwg2/heJQ8BNWcEt75SpabZHMfr064n82mlcc7VoMiUmyldnsHMbetS6gUDc1mVy7mtqyAyfcB9352E0EaaeCto0Jura6g73bqOK57fi9l1CCNy0wblCvlnZ1Urthcavi2xMO7ZrB+wWv58KQkJSx8SUedDUJimQi1BUqGw+iDKnt8AEEi2Ia/7GHCBvdAYou2bEELRkotgA8Cl7TEEYJpq9FNKGTFKSm2brkKZsmX0VlGwSqEKTrvjGpV2QzVypxiuyS4wa3+J1EIgAKHAnAKnCuy1t5kUlWdSr9TVF6njH90QPCog3sdSOKEq1sqfuGOki7YX1T+eKWE10TdLPin6KBl2RsN63S3CaslGEGSO6Dexqf9tHNvG4lfPREZAWh1novW7YNcvXbpstpet2iB5Mcxz6kVlURqAQHOFe8ASLcQZoJgbnMXXgnlFTPMXBEoAEzIvuC25zMqCR8cZVGLrb5mlDrltI2r/RgKwXCtk1RbatZ6q7UCvtfcaeALbldtNLOnFgFteuPmq7ITgMuNr/xTOwQBnqrH4aUll0yTgwawrCBSQSbou54xYPEicmyZ/iQLLTBTwOF/wMLW9B54uD7imhFtKoBhwiQ+YckacYicoKrMMAQgkCQ+ZcX2+1kVU2ndqLSzLAmLCFEKJ+5fxO09z9UEkLnVFAooVQWpJ5IwcIjgwAhOCabdaCs+XW2ddqJBRq8gutLOhqwqBqU9BBUONRCJ55+O3OeDQyABKf196i0VcQAkpqYKScX2+tT4wEJmHvKy1Jn1c0Ce0/S30+vfxKZzld0crodjBK613NsKi08VOaNAvoh2leOAL3i2i+6UMoStghBm1Y0eF0ki5aIX0Z04LylV57aE06Cs2teGi4qyZb66MOzOL5skeP1dtWTK9ElGJZmqasacxVDh6G8cF7h5tF+H7vKcmI9bY9FiT7K/vnlTa62PZ9ZiFa7QMQBjSVioLLdc7iOMUu98+mmjIP11d96J6VMD4NuacRcAYKEvbpX2iG/OohaPt0nBUv/jOps8IAdXasX26RX6hm+1j/diFd3a19tb783/tWPbj2j7nCB0WCu/liCPaMIE2yA50X6fx8XcUCCeKHujUQyFmMIB9QTLAxk/Ri7vl3otyWpO1LHh0jasK6REbUaTMveRz5fKbcyk/AGR8B1wiT6qp3gsZr2nt0Ymh+UI6X/pKGOiQce0cRbd4pu8XRgEt0kYKEm0dQFkDImtZJDKJiiFjGJxpTwwtB1UnaLK8J64wyLptvj2rXjM+hlFUEhFh0lTXhayl4Nd1jNo/ym+kMFemcZSUb4uv64i0zZrmwr83W74X2KPjr+XVP10ovJipnSjzzDPOtJNACD4n83atnFXRsK6eV5YJmMYawpbmcIqYMNqJbEQVWtl3wtRryjTQuwsktCq1XsNq85aCKpLDsjWhFZwEQgxADABRBmXZ3Q0hCqzACZzVKWlSbghXqs/Z9gOgSq7GFLmun3jpHKj476D9WzQawyMnq1pSXZ3N/SOBAKBjqN7xqZCQ+mpUGOQSFTPFFnKqPo9QPjqEiQgIsYcx0KeXIDv2C9n9nkfacOvT5ny2EJFNkGfXJyhcZPdLsOG7NkWELds+s0E5hJR6B7i9b/QutRwrjPSvRkVpunP7zmwf3HuOtcK2rMcj9POjj4BT8FFnfh6gsx1zRHv8ETSa4P8EqkykNr9pnwSqaZyJGbnEb4cItIR8LSqDi4TZep8aEQMUxm0gu48yDgCsJrpnKmu4aQw9KFm83JZvGU+L15d7lFmNGJIVELYOI6amjDil1CWj2xrvexbxFkSSc65pT+zKaA03VZhGGfL1eu32ZLbRP8xc294gt+L3coLDQmp7cF5vGTTfgfT3OtPCnhDwfW2f7RWBo/QulsIpzVz/PwEfnWvkCYljavQjqMZob1TDmuLtRZfrf5SgsFbNmVug99HguCGrLSoSYkwnQjFsihWSkzgdiVn82ASACFQWJBAymHKFjiQ5n9xdDTGyY7RYOAOYW7VZe8+PERQWIujPWA2watR6T6nqCoJhheDQjrUCO6gohCD7YROJlVAstsy5GltkXqs8k7u3XMcnhTq1Q6Du3hq15hjeHu/wGrRlsB7ygWGSvt+IbCbVXDF8YL3or4ukGnyUrN9iS2MfWQg2w2trm10PYa31etR8eoCTFUFlHxl1fNx+DEvhBJ3XoO+b6uN7jtALGbOFiYDVrmAAutRMypjqwEGbVC8jf+89KOlEH3ZMnqApukW26HMS7DoRy6CZSwFc4sKJAISi+UXk5VY24xF4IhRoJ0AT4km+JOJsWBVJlJdOEjNbJMLLYOGumd6n8DKr7SVjsE+ZrM8eMRuCYUTMQOYS1abQm0A8GU2IcMo1T1SA7m7NAgNRQADhljIyJ2RO0JQREiXU6iTCMhUmHJBNVI46/GXfhNIWYki4ZQJonbba+gC0nav2Oh6wLEsXMcTl/Y6YtwoEm/vJbhLkHddWKOj5QC1RnhUgtr7+nC9ThZFP+y1WzTZbZpt+vihsKgCAXpjpdSqQj9LHEApnlXm9bce0/B3pLHT2q5HiteVXUfYZXCEBBgcJMxShULBwBiIkeRkyg6mY4wQw8SrTpU4gwEBIW8Yqc7FURnX9GOSZ5F7d9Fqf7sGet5qsTfkwgom0LGWmFrayq5PPtOPovLZ19P4BeaF9ugmf3M7meVK4zOdq8szeM/qRUPDW5MhaUD+CtRbK1d0WoKP3qcLMRt/ZOtvsr9ruLnjgDh0XCu82CXQgvG2pI63i55JfmSxkdC53dV9vP/Hfzuew0T984JqN4w39sWbDBq5vOoBpRuXONJl7yoI0CgDK+g0S12ZmEsODAC6x7mAgpQwOZPZGlkRkrV8Vb83FcKBS9ggiYnPNe1KbC6NXWrwlUPuFStRVq+MYx7Z/9fuIoXmoRS0PYlllrr3XhHe7hotvo8IoINlXG2K5cMoIMRQjg2t593wFHh/3x6wjGTDhn4Ndxqxj2P4d+Sbs82sKFaOBa5ZYT3sZef078H3NzGKthbUw0O82MslaA16gWWGodd+rm6UTQuHwlTKnD06gl8AgWwzSduLHsSIKdEA98qrEesnotp0J83pBcO/99NeM3+e6nK5WB6KchKeUiRhmUAn14zA3TYHLSlYAVI4LVKRCIkACZArUBEJKLSy1Mq6aKMyOn1yYsDBb76irjXoDeXB/TpTFe7ytKsj/oagYGp3VO5uHzB29FTDSgpVpdCGRLE+1+wcIlGn2WiYqq6ozsoVCyjaUzLIiPeqK5JQRKZR0Js1asz4BzxS9f8AeU61fGd9oDUbt4QIb2XxG3WrlDV+EhZE4M3K6riKVbCRU7aoBcx9p/t7K2rL4vC/C1tOvX6lOd1rvsbBHHwM+OkEfR/P/pLcgmYwlJ87lC2KU6A9MDzLhjCOYASS0latshAOR5N2hGAGNnIFxRg5FcptMikj9KqSMyENAI+enP2/DPgFUZuqhiG3h1IeO+rxCwHonNV8fZagj5mfDZb0WvyUobN6mEIL4SwhD2ISorVzWOmuKDk3vYcNZtX+atk3gnFYRS7b/LOPWtOOj92QZdhXavHTttMJa+1T3iLBl+f7WvwpTvbml8FYa9+gFnaGRlD3zvDP33qMhk6l//Vly08ucH1RDYvR3nr0zWY7QfQvthRyyMJIG0wxCI8kwghBBYQKFCWF6QpguiNMMmp8kb05xKGt9bpYHKqRBAlXEGMUJHQS2yiBkUFtoVerXnLKQa1HGgmtyraOiUpt9vA4FXF2xOQ7bBizlqQNG2Rbiab2sZki0rrt9roUa7Hc9b52u9nhKCZnFCpnneRVjbxmaPW41Zo3sGWneYoGNU0177Vavsfi7Mneg37NZfBwTggl59aSJ/GKMdatOFQyjNQt9PxJy6n0E6lfRe7UvdP9q37faPivYqlBJ4322bd3nea4hvnZzoS3LRN/nEXoXobA3QewAeSm9NyR0uH4b9eDi5BpfSubv3nO2tbRzTGdN+XD/nX9PrVX6jIx1O9okI4qgMINCBMUHhOkBcVahEEEUQMZRrAuoAHQTtjI3wS4LpIEWhWP7h6vrocFShmn1fSnvgXd8CuoD8GXcU4C4YIe747kMA73WCo3KALrNidbPswzbQy/AOoXFCqpxvy2WrQ5aKzDs4jfv9FQSS07qbgMB7HOsgLBM2VooVnj40E6/N4Ntfwiyc5um8dbcR14o2Do3DVz8ULbP1EqpacFLG799+zYUCpaxeye+h5G80LD+kBFs5HmvVwTu0bvAR/cY/1sIhveij1qvX5WGDK90sWp6cboU+GjGNMknFww7cwYvt6pNJ8aKcXVQClEL8SUJY2X2ezSzrcSOQPgY5Cf+6Lz97rVFr6X7334VrSWN9LLlKsO10MyIIVkmDaDDwIFq93TWg4dSRu9ZI530+TnnLqPrNE2IU+wY56jftB3WIlEIyTNrYA3HWejK5kzStmj7NcrJ96tPyld7hdaCwfaJ/diQ1rdSlt/dpzDqRD3+syfga5+fhykcxL/aR91sOYfZXLM+dUa6jzTTbWKY3W3Gz+8slSPXtEsjCzzVkHz9q/0QAIoIMSBMXxBnMeNBj2C6ICNiQaxpkW9sVt5yibkmjXApi6JSKjAP1aqGKIumQnFwgiF5d4qEkO5aO+D6MYsqQ7bG7LHJ2Du41/3YdyIhVI0U6DVkn68olRAuC1v4ulqhopCLD18cMXhdGVykMnS1uGY/BQiZJVswMco6B1kXkrMsNETJnZSWJO+DAlJOBTocM+0RrKJtsvi+tSL0uGj7PfQDNKGkZdQ3Y75vOaC1HqzjKPT7UagloOsP9Hv3lp2A1GtsfRgMybo7DnW1v+3HPmts3RwXGj8UPhqZNfae96AtDeHtyh8c65+28X3rmp2C34U6FboRmdMnyIoAoEEr1otRUqqBaEKYLgjxUtIuT2BEpBJNlFmgrsVOjFIWFW4datTQGhYhCmBd9lwECljhItvIUscB4+eCe4P7cb3bB4PxPRas98oiaJSPlreCjupz1uf2tPgtAdPaXWAW25/QVcnC2Lt5jsaA66K5zFWwAwCnBHAAgsCABFnUtlXvLehZmb/9bqEi+R46KxAYh4tqnRXCGkGSI6GgPNgKVGsh+LEy+u5DSpXU0ewtBX//EUTmJfQuIakMU9laMWp/bGWHE+iNyc9NU6UulUR15B0vdnVtKZ+7q8qpzRe49cCjffJCDn7oOUfL3h6cMkH6EhQGmKcJ0yxhphmSGhspYSlvI+vK5MKwcyopLExfEqEyK69hdpOTGalEpVgsWtunoZHdWKRqWAyZ526PvET5KI/2juGRIKjVZLHJ7GIzLcPH8Guf6DkbLeQjhlapIwr5iJ4RAx3V20JE0td9plItY7R+wL4Tny/Jw0BUBgSjHw9qSdmFe1quX+BoyfcBM8sWsea83wNa+3glUAxtIQG6lWb5gczbFod3bus9OxjEXTqe++hEoaOdLqnOrHWZhxON4gXCwxdul6f6paosmtlRSolk8dQOEdnBvLXcfNSmATSzSYz7qSr2nrf3nDNlq1lQNCBI6gNmBiMi68Y60x+g6QnT4yN4+hMpimOOaWr4NRp0Eo1zflHznVvyMHVYxhCAoiGGKFlTM5f91uoahlLPjde2Gl+l+eSueXvLts2NPS2xndfvRbFRLdts3ZhTRs4JXARpVAcqCuSk6TCIEClgCi3hG5WNb8igXmqdMavwlOeEIGO1RkIhlfezwK5nkHvV1wDQBvzantWTX6A2uqcKASKA1umqVYB4fN5HUdn34IUkM2RtJdBZLV4gKNm1EyNBua7frK+w/GGAZQdBffF611zCZ1NKgOapAheY1PajXw67Te8iFAB0KrRgkoA2pXvhfD848nU0qrmDBwydkTmMmmZt8wr0Su2gPq+1Euz17wDB0YGyXVW9baGQAiEWreqC6fKI+fKIMM2gUGcYVLAQ91FLBDIeCsi4Ua245OIhLQJUBDwXphRARZvWCnYWYsGOyFgDWm+tA/MYXmnXbnfPOWoM3j5rhA0Lj1D4xtRJz+dcwxulb0JdhNbtjUBUP8yy2pgBxBAlh0jXL9InwnyBnPsMncGG+Mq3TtFqfpJSmrMI9sji6v4ebx2CqGxoP84Y6xPf2bUA3irwzFtHhS1j1JZ7OL4X/LVt0Da4a4uVaymWNOYqL3KGzJ16HbX/D47RH5Il1U80b573cMvb1sVjx3t07yUO7hgKBeq+ccdSx+VvPPNgXV6mtW6VfQTiundNaTERlNkSEQIC5vmC+eEB0+UBDw8PQJyBIhSYpgYZZWEgWWVDx2QKFJGSsmxovDvQkpplzkBWy0NrrZbG2lryzEYEDkSjDuuFU/7e9fEGnZ2lkbUwgo9yziUJSN8mC8tYzJ2IOpjJQza2DSEEUBT8zJal2nbHUJnB1EJUVRBtWTojh+9eX2xF/mxRCAHB1MU70+33ETzk+9yG66olZfMl+XusE9/CefYZo2dt+VNGcJ19d339MCznKJ/4ECua39dSOEdnhELGhHEX5tV34Wst6VXPQKud2NjtCRjrPJxhn2/pDQRC3QKTofAcUcQ0PQHTH3h6ekJ8+HfEy5+yopRiyWkE5HyTFM0ZoJTAOYnqY7d5LhquQB8tRbZaCigWgUQ4QaKTOIM4NQgTE/ajgWyzuESTHWdIb0Ee3hAGo797K0Etp9G9yjiUidtoI6BBGxan73MDicBNmYu1Vu6Drr4txgQFhEi4ZRbIEIyUU90XWokhM2IKEXEgiDz59RVHSFfJe6fxPa3d+6RGPgApu7/PCwYNUb1er3h+fq5bf+r1XmDbZ/fP6evj+0nbZIWPt6bOWGJKP18oWKv0Rz96x6I5QgRN7eyPhzZJSROYKQRxtHKnqnKOjvD+l5bH9mvDvokiwjTj8vCI+PCIOD9gmmdkBGSoNp8EEzeTJudcZawIVmPO6welX8vvrWukEDaupJcNui2rYNUtrxAenjG1NjnYqEKz63v0mDJ4u4eAnvNWwjoPUF9OZeKQyLDscP4FLR6fTB26tm3ALVtk67x3jbfeumcOnrPu2/W6iD2FyzJ4K7g0LFXTS1jH9ug+Tz6V9paVYNvl17F4gXiG3m07zqMM9t2tBGohcsB5xn+n6D5RmDxAgg+p4dEgOgwF/a5EgWpKgThfEGq+mIgATalQBjgzci6TKeViQTUYJqu1UMnAJ9yESjRQU71GLnz1ODhvnR0nX24vENbMzoZ9+vs807cMyqZysOX1jKiV66EeC6vY59zr2yZYxvsgr9t9bEXuSLPXsq214GE2vdYzW9+WrTr6sFS7veZo0eFWHWwZPnLJC3zr4B6V6eG5U5A7vyWX/KRP+qRP+qRfms7EMX7SJ33SJ33Sb06fQuGTPumTPumTKn0KhU/6pE/6pE+q9CkUPumTPumTPqnSp1D4pE/6pE/6pEqfQuGTPumTPumTKn0KhU/6pE/6pE+q9CkUPumTPumTPqnSp1D4pE/6pE/6pEr/P7uucIfz2brqAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Captions: ['a diagram', 'a dog', 'a cat']\n", + "Label probs: 0.00 1.00 0.00\n", + "This video is about a dog\n" + ] + } + ], + "source": [ + "video, first_frame = preprocess_video(\"./apps/pe/docs/assets/dog.mp4\", 8, transform=preprocess)\n", + "video = video.unsqueeze(0).to(device)\n", + "text = tokenizer([\"a diagram\", \"a dog\", \"a cat\"]).to(device)\n", + "\n", + "with torch.no_grad():\n", + " image_features = model.encode_video(video)\n", + " text_features = model.encode_text(text)\n", + " image_features /= image_features.norm(dim=-1, keepdim=True)\n", + " text_features /= text_features.norm(dim=-1, keepdim=True)\n", + " text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1).cpu().numpy()[0]\n", + "\n", + "plt.imshow(Image.fromarray(first_frame))\n", + "plt.axis('off')\n", + "plt.show()\n", + "print(\"Captions:\", captions)\n", + "print(\"Label probs:\", ' '.join(['{:.2f}'.format(prob) for prob in text_probs])) # prints: [[0.00, 1.00, 0.00]]\n", + "print(f\"This video is about {captions[text_probs.argmax()]}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "aceb0cab", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.20" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/perception_models/apps/plm/README.md b/perception_models/apps/plm/README.md new file mode 100644 index 0000000000000000000000000000000000000000..7074c05da377bc6e8010521ece8213eff2e08fd5 --- /dev/null +++ b/perception_models/apps/plm/README.md @@ -0,0 +1,99 @@ +# Perception Language Model (PLM) +[![Paper](https://img.shields.io/badge/Paper-PerceptionLM-b31b1b.svg)](https://ai.meta.com/research/publications/perceptionlm-open-access-data-and-models-for-detailed-visual-understanding) +[![Paper](https://img.shields.io/badge/arXiv-2504.13180-brightgreen.svg?style=flat-square)](https://arxiv.org/abs/2504.13180) +[![Hugging Face Collection](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Collection-blue)](https://huggingface.co/collections/facebook/perception-lm-67f9783f171948c383ee7498) +[![Colab](https://img.shields.io/badge/Google%20Colab-Tutorials-red)](notebook_demos) +[![ModelLicense](https://img.shields.io/badge/Model_License-FAIR_Research_License-lightgrey)](../../LICENSE.PLM) + +--- + +This is the official implementation of **Perception Language Model** from our paper: +**[PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding](https://ai.meta.com/research/publications/perceptionlm-open-access-data-and-models-for-detailed-visual-understanding)** +Jang Hyun Cho*, Andrea Madotto*, Effrosyni Mavroudi*, Triantafyllos Afouras*, Tushar Nagarajan*, Muhammad Maaz*, Yale Song*, Tengyu Ma*, Shuming Hu*, Suyog Jain, Miguel Martin, Huiyu Wang, Hanoona Rasheed, Peize Sun, Po-Yao Huang, Daniel Bolya, Nikhila Ravi, Shashank Jain, Tammy Stark, Shane Moon, Babak Damavandi, Vivian Lee, Andrew Westbury, Salman Khan, Philipp Krähenbühl, Piotr Dollár, Lorenzo Torresani*, Kristen Grauman*, Christoph Feichtenhofer* +\* Joint First Author / Project Lead + +_[HuggingFace](https://huggingface.co/collections/facebook/perception-lm-67f9783f171948c383ee7498)_ | _[Blog](https://ai.meta.com/blog/meta-fair-updates-perception-localization-reasoning)_ | _[GitHub](https://github.com/facebookresearch/perception_models)_ | _[arXiv](https://arxiv.org/abs/2504.13180)_ | _[BibTeX](#citation)_ + +![Description of the image](docs/plm_main_fig.png) + +PLM consists of a vision encoder with a small scale (<8B parameters) LLM decoder. We start by an analysis of standard training pipelines with available data, without any proprietary model distillation. We investigate large-scale +synthetic data and establish key scaling laws to identify critical data gaps that limit video understanding performance, especially for spatio-temporal reasoning and fine-grained understanding tasks. To fill these gaps, we release 2.8M human-labeled instances of fine-grained video question-answer pairs and spatio-temporally grounded video captions. This release is nearly an order of magnitude larger than the largest existing video datasets. Additionally, we introduce PLM–VideoBench, a suite for evaluating challenging video understanding tasks focusing on the ability to reason about "what", "where", "when", and "how" of a video. + + +## PLM Resources + +| Resource | Description | Documentation | +| --- | --- | --- | +| **Evaluation** | Evaluation of PLM using lmms-eval | [`docs/evaluation.md`](docs/evaluation.md) | +| **Training / Finetuning** | Training and finetuning instructions for PLM | [`docs/training.md`](docs/training.md) | +| **PLM-VideoBench** | Evaluation on PLM-VideoBench using lmms-eval | [`docs/plm_videobench.md`](docs/plm_videobench.md) | +| **End-to-End Finetuning Example** | End-to-end finetuning example on radiology images | [`docs/finetune_example.md`](docs/finetune_example.md) | +| **Generating Response** | Generate responses using a trained model with `generate.py` | [`generate.py`](generate.py) | + + +> [!TIP] +> To run the following code, download the [`dummy-datasets`](https://dl.fbaipublicfiles.com/plm/dummy_datasets.tar.gz) and extract them to `apps/plm/dummy_datasets`. + +```shell +python apps/plm/generate.py \ +--ckpt facebook/Perception-LM-3B \ +--media_type image \ +--media_path apps/plm/dummy_datasets/image/images/14496_0.PNG \ +--question 'Describe the bar plot in the image.' + +# Expected output +The image presents a bar graph with four distinct categories: step, horror, mood, and lumber. The x-axis lists these categories, while the y-axis is labeled "Values" and ranges from 0 to 10 in increments of 2. +... +... + +``` + +```shell +python apps/plm/generate.py \ +--ckpt facebook/Perception-LM-8B \ +--media_type video \ +--media_path apps/plm/dummy_datasets/video/videos/GUWR5TyiY-M_000012_000022.mp4 \ +--question 'What is happening in the video?' + +# Expected output +A group of individuals are skipping rope in a coordinated routine on a basketball court. They are standing in a line, with each person holding a rope and performing a synchronized movement, with their arms extended and their bodies in motion. The court has a blue center circle and white lines marking the playing area, and spectators are seated on the sidelines watching the performance. +``` + +## Tutorials + +For more task-specific usecases, check out our notebook tutorials: +- [Image and video captioning](./notebook_demos/image_and_video_captioning.ipynb) +- [Image grounding and region captioning](./notebook_demos/image_grounding.ipynb) + +## PLM Image Benchmark Results + +| Model | DocVQA | ChartQA | TextVQA | InfoQA | AI2D | OCRBench | COCO | Nocap | Flickr | MMMU | VQAv2 | OKVQA | VizWiz | MME | SEED | BLINK | CVBench | RealWorldQA | VSR | POPE | +|:---------:|:--------:|:---------:|:---------:|:--------:|:------:|:----------:|:------------:|:-------------:|:--------------:|:------:|:-------:|:--------:|:--------:|:-----:|:------:|:-------:|:----------:|:-------------:|:-----:|:------:| +| PLM1B | 90.7 | 78.6 | 82.1 | 63.0 | 84.9 | 807 | 138.6 | 124.2 | 100.5 | 34.8 | 81.7 | 61.0 | 59.7 | 1603| 76.3 | 46.8 | 73.8 | 67.1 | 68.8| 88.4 | +| PLM3B | 93.8 | 84.3 | 84.3 | 74.6 | 90.9 | 830 | 144.9 | 126.5 | 98.0 | 41.2 | 84.3 | 66.8 | 64.0 | 1879| 78.5 | 55.4 | 81.4 | 72.4 | 80.4| 88.7 | +| PLM8B | 94.6 | 85.5 | 86.5 | 80.9 | 92.7 | 870 | 146.7 | 129.9 | 105.6 | 46.1 | 85.6 | 69.6 | 67.0 | 1989| 79.3 | 56.0 | 81.3 | 75.0 | 82.8| 89.9 | + +## PLM Video Benchmark Results + +| Model | VATEX | DREAM 1K | How2QA | MVBench | NExTQA | PerceptionTest (test) | STAR | TVQA | VideoMME | TVBench | ActivityNetQA | EgoSchema (test) | TemporalBench | TOMATO | MotionBench (dev) | TempCompass (MCQ) | CGBench (clue) | Charades STA | VideoHallucer | Halluc. EventHallusion | +|:-------------:|:---------------------------:|:-----------------------:|:---------------------:|:-------------:|:-------------:|:--------------------------:|:----------:|:----------:|:----------------:|:-------------:|:--------------------:|:----------------------:|:---------------------:|:------------:|:------------------------:|:-----------------------:|:---------------------:|:-------------------:|:-------------------------------:|:--------------------------------:| +| PLM1B | 92.5 | 34.3 | 86.4 | 70.1 | 80.3 | 72.7 | 83.7 | 50.3 | 49.2 | 50.4 | 62.5 | 60.4 | 18.2 | 25.5 | 52.2 | 64.6 | 43.6 | 55.2 | 49.2 | 79.5 | +| PLM3B | 96.1 | 37.4 | 89.4 | 74.7 | 83.4 | 79.3 | 84.8 | 55.3 | 54.9 | 58.9 | 66.2 | 66.9 | 23.4 | 30.9 | 60.4 | 69.3 | 47.2 | 57.7 | 55.5 | 76.5 | +| PLM8B | 99.7 | 35.9 | 90.7 | 77.1 | 84.1 | 82.7 | 84.9 | 59.3 | 58.3 | 63.5 | 67.3 | 68.8 | 28.3 | 33.2 | 61.4 | 72.7 | 46.4 | 58.6 | 57.7 | 77.3 | + +## Citation +```BibTeX +@article{cho2025PerceptionLM, + title={PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding}, + author={Jang Hyun Cho and Andrea Madotto and Effrosyni Mavroudi and Triantafyllos Afouras and Tushar Nagarajan and Muhammad Maaz and Yale Song and Tengyu Ma and Shuming Hu and Hanoona Rasheed and Peize Sun and Po-Yao Huang and Daniel Bolya and Suyog Jain and Miguel Martin and Huiyu Wang and Nikhila Ravi and Shashank Jain and Temmy Stark and Shane Moon and Babak Damavandi and Vivian Lee and Andrew Westbury and Salman Khan and Philipp Kr\"{a}henb\"{u}hl and Piotr Doll{\'a}r and Lorenzo Torresani and Kristen Grauman and Christoph Feichtenhofer}, + journal={arXiv:2504.13180}, + year={2025} +} + +@article{bolya2025PerceptionEncoder, + title={Perception Encoder: The best visual embeddings are not at the output of the network}, + author={Daniel Bolya and Po-Yao Huang and Peize Sun and Jang Hyun Cho and Andrea Madotto and Chen Wei and Tengyu Ma and Jiale Zhi and Jathushan Rajasegaran and Hanoona Rasheed and Junke Wang and Marco Monteiro and Hu Xu and Shiyu Dong and Nikhila Ravi and Daniel Li and Piotr Doll{\'a}r and Christoph Feichtenhofer}, + journal={arXiv:2504.13181}, + year={2025} +} +``` diff --git a/perception_models/apps/plm/configs/datasets.yaml b/perception_models/apps/plm/configs/datasets.yaml new file mode 100644 index 0000000000000000000000000000000000000000..09611eb758d9dc44aab850d67efb817fa6be785d --- /dev/null +++ b/perception_models/apps/plm/configs/datasets.yaml @@ -0,0 +1,30 @@ +dummy_image: + annotation: apps/plm/dummy_datasets/image/annotations.jsonl + root_dir: apps/plm/dummy_datasets/image/images + +dummy_multi_image: + annotation: apps/plm/dummy_datasets/multi_image/annotations.jsonl + root_dir: apps/plm/dummy_datasets/multi_image/images + +dummy_image_region: + annotation: apps/plm/dummy_datasets/image_region/annotations.jsonl + root_dir: apps/plm/dummy_datasets/image_region/images + +dummy_video: + annotation: apps/plm/dummy_datasets/video/annotations.jsonl + root_dir: apps/plm/dummy_datasets/video/videos + +dummy_text: + annotation: apps/plm/dummy_datasets/text/annotations.jsonl + +dummy_stc_RDCap: + annotation: apps/plm/dummy_datasets/plm_stc/RDCap.jsonl + root_dir: apps/plm/dummy_datasets/plm_stc/videos + +dummy_stc_RCap: + annotation: apps/plm/dummy_datasets/plm_stc/RCap.jsonl + root_dir: apps/plm/dummy_datasets/plm_stc/videos + +dummy_stc_RTLoc: + annotation: apps/plm/dummy_datasets/plm_stc/RTLoc.jsonl + root_dir: apps/plm/dummy_datasets/plm_stc/videos diff --git a/perception_models/apps/plm/configs/stage_1/plm_1b.yaml b/perception_models/apps/plm/configs/stage_1/plm_1b.yaml new file mode 100644 index 0000000000000000000000000000000000000000..f1090cba533353fd6f30bd67baa8121f2e1519c6 --- /dev/null +++ b/perception_models/apps/plm/configs/stage_1/plm_1b.yaml @@ -0,0 +1,81 @@ +# We use a global batch size of 512 in stage # 1 for PLM-1B model. Please adjust batch_size as per your training setup. +# For example, one possible configuration is batch_size=16,nodes=4,gpus_per_node=8 = 16*4*8 = 512 global batch size. + +name: "plm_1b_stage1" +dump_dir: ./plm_1b_stage1 +steps: 8000 +seed: 777 +optim: + lr: 1e-4 + warmup: 20 + lr_min_ratio: 0.01 + clip: 1.0 + weight_decay: 0.01 + +distributed: + fsdp_type: full_shard + compile: false + model_dtype: bf16 + matmul_allow_tf32: false + selective_activation_checkpointing: false + full_activation_checkpointing: true + tp_size: 1 + +model: + dim: 2048 + n_layers: 16 + n_heads: 32 + n_kv_heads: 8 + vocab_size: 128256 + ffn_dim_multiplier: 1.5 + multiple_of: 256 + norm_eps: 1e-05 + rope_theta: 500000.0 + weight_tying: true + rope_scale_factor: 32 + high_freq_factor: 4 + max_seqlen: 1280 + freeze_language_model: true + freeze_vision_model: true + pooling_ratio: 1 + vision_model: + image_size: 448 + patch_size: 14 + width: 1024 + layers: 23 + heads: 16 + use_cls_token: true + use_abs_posemb: true + mlp_ratio: 4.0 + use_ln_post: false + pool_type: "none" + mlp_init: + use_gaussian: true + +data: + datamix: + num_workers: 8 + batch_size: 16 + image_res: 448 + max_num_tiles: 1 + max_video_frames: 8 + vision_input_type: vanilla + tokenizer_path: facebook/Perception-LM-1B/tokenizer.model + tokenizer_name: plmchat + conversation_format: warmup + +profiling: + run: false + +checkpoint: + dump: + every: 500 + keep: 1 + init_ckpt_path: meta-llama/Llama-3.2-1B-Instruct/original + # Please use the script at apps/plm/interpolate_PE_pos_embed.py to interpolate PE-Core-L14-336 (https://huggingface.co/facebook/PE-Core-L14-336) checkpoints to 448 resolution. + vision_model_path: facebook/PE-Core-L14-336-interpolated-to-448/model.pt + is_consolidated_model: True + +logging: + freq: 10 + level: INFO # Available choices for logging level are: [NOTSET, DEBUG, INFO, WARN, ERROR, FATAL, CRITICAL] diff --git a/perception_models/apps/plm/configs/stage_1/plm_3b.yaml b/perception_models/apps/plm/configs/stage_1/plm_3b.yaml new file mode 100644 index 0000000000000000000000000000000000000000..182d0e1239c58cb377d20bbdebea61f20e825052 --- /dev/null +++ b/perception_models/apps/plm/configs/stage_1/plm_3b.yaml @@ -0,0 +1,81 @@ +# We use a global batch size of 512 in stage # 1 for PLM-3B model. Please adjust batch_size as per your training setup. +# For example, one possible configuration is batch_size=16,nodes=4,gpus_per_node=8 = 16*4*8 = 512 global batch size. + +name: "plm_3b_stage1" +dump_dir: ./plm_3b_stage1 +steps: 8000 +seed: 777 +optim: + lr: 1e-4 + warmup: 20 + lr_min_ratio: 0.01 + clip: 1.0 + weight_decay: 0.01 + +distributed: + fsdp_type: full_shard + compile: false + model_dtype: bf16 + matmul_allow_tf32: false + selective_activation_checkpointing: false + full_activation_checkpointing: true + tp_size: 1 + +model: + dim: 3072 + n_layers: 28 + n_heads: 24 + n_kv_heads: 8 + vocab_size: 128256 + ffn_dim_multiplier: 1.0 + multiple_of: 256 + norm_eps: 1e-05 + rope_theta: 500000.0 + weight_tying: true + rope_scale_factor: 32 + high_freq_factor: 4 + max_seqlen: 1280 + freeze_language_model: true + freeze_vision_model: true + pooling_ratio: 1 + vision_model: + image_size: 448 + patch_size: 14 + width: 1024 + layers: 23 + heads: 16 + use_cls_token: true + use_abs_posemb: true + mlp_ratio: 4.0 + use_ln_post: false + pool_type: "none" + mlp_init: + use_gaussian: true + +data: + datamix: + num_workers: 8 + batch_size: 16 + image_res: 448 + max_num_tiles: 1 + max_video_frames: 8 + vision_input_type: vanilla + tokenizer_path: facebook/Perception-LM-3B/tokenizer.model + tokenizer_name: plmchat + conversation_format: warmup + +profiling: + run: false + +checkpoint: + dump: + every: 500 + keep: 1 + init_ckpt_path: meta-llama/Llama-3.2-3B-Instruct/original + # Please use the script at apps/plm/interpolate_PE_pos_embed.py to interpolate PE-Core-L14-336 (https://huggingface.co/facebook/PE-Core-L14-336) checkpoints to 448 resolution. + vision_model_path: facebook/PE-Core-L14-336-interpolated-to-448/model.pt + is_consolidated_model: True + +logging: + freq: 10 + level: INFO # Available choices for logging level are: [NOTSET, DEBUG, INFO, WARN, ERROR, FATAL, CRITICAL] diff --git a/perception_models/apps/plm/configs/stage_1/plm_8b.yaml b/perception_models/apps/plm/configs/stage_1/plm_8b.yaml new file mode 100644 index 0000000000000000000000000000000000000000..f2bbda7152fbbcd7c2a70ae467d280b833e1ba25 --- /dev/null +++ b/perception_models/apps/plm/configs/stage_1/plm_8b.yaml @@ -0,0 +1,78 @@ +# We use a global batch size of 512 in stage # 1 for PLM-8B model. Please adjust batch_size as per your training setup. +# For example, one possible configuration is batch_size=16,nodes=4,gpus_per_node=8 = 16*4*8 = 512 global batch size. + +name: "plm_8b_stage1" +dump_dir: ./plm_8b_stage1 +steps: 8000 +seed: 777 +optim: + lr: 1e-4 + warmup: 20 + lr_min_ratio: 0.01 + clip: 1.0 + weight_decay: 0.05 + +distributed: + fsdp_type: full_shard + compile: false + model_dtype: bf16 + matmul_allow_tf32: false + selective_activation_checkpointing: false + full_activation_checkpointing: true + tp_size: 1 + +model: + dim: 4096 + n_layers: 32 + n_heads: 32 + n_kv_heads: 8 + vocab_size: 128256 + ffn_dim_multiplier: 1.3 + multiple_of: 1024 + norm_eps: 1e-05 + rope_theta: 500000.0 + weight_tying: false + max_seqlen: 1280 + freeze_language_model: true + freeze_vision_model: true + pooling_ratio: 1 + vision_model: + image_size: 448 + patch_size: 14 + width: 1536 + layers: 47 + heads: 16 + use_cls_token: false + use_abs_posemb: true + mlp_ratio: 5.833333334 + use_ln_post: false + pool_type: "none" + mlp_init: + use_gaussian: true + +data: + datamix: + num_workers: 8 + batch_size: 16 + image_res: 448 + max_num_tiles: 1 + max_video_frames: 8 + vision_input_type: vanilla + tokenizer_path: facebook/Perception-LM-8B/tokenizer.model + tokenizer_name: plmchat + conversation_format: warmup + +profiling: + run: false + +checkpoint: + dump: + every: 500 + keep: 1 + init_ckpt_path: meta-llama/Llama-3.1-8B-Instruct/original + vision_model_path: facebook/PE-Core-G14-448/model.pt + is_consolidated_model: True + +logging: + freq: 10 + level: INFO # Available choices for logging level are: [NOTSET, DEBUG, INFO, WARN, ERROR, FATAL, CRITICAL] diff --git a/perception_models/apps/plm/configs/stage_2/plm_1b.yaml b/perception_models/apps/plm/configs/stage_2/plm_1b.yaml new file mode 100644 index 0000000000000000000000000000000000000000..3c907f3cd2ad7a3147aa0b8eaa3b34575bd11a71 --- /dev/null +++ b/perception_models/apps/plm/configs/stage_2/plm_1b.yaml @@ -0,0 +1,81 @@ +# We use a global batch size of 2048 in stage # 3 for PLM-1B model. Please adjust batch_size as per your training setup. +# For example, one possible configuration is batch_size=8,nodes=32,gpus_per_node=8 = 8*32*8 = 2048 global batch size. + +name: "plm_1b_stage2" +dump_dir: ./plm_1b_stage2 +steps: 35000 +seed: 777 +optim: + lr: 4e-5 + warmup: 120 + lr_min_ratio: 0.01 + clip: 1.0 + weight_decay: 0.01 + +distributed: + fsdp_type: full_shard + compile: false + model_dtype: bf16 + matmul_allow_tf32: false + selective_activation_checkpointing: false + full_activation_checkpointing: true + tp_size: 1 + +model: + dim: 2048 + n_layers: 16 + n_heads: 32 + n_kv_heads: 8 + vocab_size: 128256 + ffn_dim_multiplier: 1.5 + multiple_of: 256 + norm_eps: 1e-05 + rope_theta: 500000.0 + weight_tying: true + rope_scale_factor: 32 + high_freq_factor: 4 + max_seqlen: 6144 + freeze_language_model: false + freeze_vision_model: false + pooling_ratio: 2 + vision_model: + image_size: 448 + patch_size: 14 + width: 1024 + layers: 23 + heads: 16 + use_cls_token: true + use_abs_posemb: true + mlp_ratio: 4.0 + ls_init_value: 0.1 + drop_path: 0.1 + use_ln_post: false + pool_type: "none" + mlp_init: + use_gaussian: true + +data: + datamix: + num_workers: 8 + batch_size: 4 + image_res: 448 + max_num_tiles: 16 + max_video_frames: 16 + vision_input_type: thumb+tile + tokenizer_path: facebook/Perception-LM-1B/tokenizer.model + tokenizer_name: plmchat + conversation_format: plm_sft + +profiling: + run: false + +checkpoint: + dump: + every: 500 + keep: 1 + init_ckpt_path: + is_consolidated_model: True + +logging: + freq: 10 + level: INFO # Available choices for logging level are: [NOTSET, DEBUG, INFO, WARN, ERROR, FATAL, CRITICAL] diff --git a/perception_models/apps/plm/configs/stage_2/plm_3b.yaml b/perception_models/apps/plm/configs/stage_2/plm_3b.yaml new file mode 100644 index 0000000000000000000000000000000000000000..cb781505597ae676f37fc8bffa0d29b346eca388 --- /dev/null +++ b/perception_models/apps/plm/configs/stage_2/plm_3b.yaml @@ -0,0 +1,81 @@ +# We use a global batch size of 2048 in stage # 3 for PLM-3B model. Please adjust batch_size as per your training setup. +# For example, one possible configuration is batch_size=8,nodes=32,gpus_per_node=8 = 8*32*8 = 2048 global batch size. + +name: "plm_3b_stage2" +dump_dir: ./plm_3b_stage2 +steps: 35000 +seed: 777 +optim: + lr: 4e-5 + warmup: 120 + lr_min_ratio: 0.01 + clip: 1.0 + weight_decay: 0.01 + +distributed: + fsdp_type: full_shard + compile: false + model_dtype: bf16 + matmul_allow_tf32: false + selective_activation_checkpointing: false + full_activation_checkpointing: true + tp_size: 1 + +model: + dim: 3072 + n_layers: 28 + n_heads: 24 + n_kv_heads: 8 + vocab_size: 128256 + ffn_dim_multiplier: 1.0 + multiple_of: 256 + norm_eps: 1e-05 + rope_theta: 500000.0 + weight_tying: true + rope_scale_factor: 32 + high_freq_factor: 4 + max_seqlen: 6144 + freeze_language_model: false + freeze_vision_model: false + pooling_ratio: 2 + vision_model: + image_size: 448 + patch_size: 14 + width: 1024 + layers: 23 + heads: 16 + use_cls_token: true + use_abs_posemb: true + mlp_ratio: 4.0 + ls_init_value: 0.1 + drop_path: 0.1 + use_ln_post: false + pool_type: "none" + mlp_init: + use_gaussian: true + +data: + datamix: + num_workers: 8 + batch_size: 4 + image_res: 448 + max_num_tiles: 16 + max_video_frames: 16 + vision_input_type: thumb+tile + tokenizer_path: facebook/Perception-LM-3B/tokenizer.model + tokenizer_name: plmchat + conversation_format: plm_sft + +profiling: + run: false + +checkpoint: + dump: + every: 500 + keep: 1 + init_ckpt_path: + is_consolidated_model: True + +logging: + freq: 10 + level: INFO # Available choices for logging level are: [NOTSET, DEBUG, INFO, WARN, ERROR, FATAL, CRITICAL] diff --git a/perception_models/apps/plm/configs/stage_2/plm_8b.yaml b/perception_models/apps/plm/configs/stage_2/plm_8b.yaml new file mode 100644 index 0000000000000000000000000000000000000000..f13b1f990c88bdcad841fc828f4e4d7d762679ff --- /dev/null +++ b/perception_models/apps/plm/configs/stage_2/plm_8b.yaml @@ -0,0 +1,79 @@ +# We use a global batch size of 2048 in stage # 3 for PLM-8B model. Please adjust batch_size as per your training setup. +# For example, one possible configuration is batch_size=4,nodes=64,gpus_per_node=8 = 4*64*8 = 2048 global batch size. + +name: "plm_8b_stage2" +dump_dir: ./plm_8b_stage2 +steps: 35000 +seed: 777 +optim: + lr: 4e-5 + warmup: 120 + lr_min_ratio: 0.01 + clip: 1.0 + weight_decay: 0.05 + +distributed: + fsdp_type: full_shard + compile: false + model_dtype: bf16 + matmul_allow_tf32: false + selective_activation_checkpointing: false + full_activation_checkpointing: true + tp_size: 1 + +model: + dim: 4096 + n_layers: 32 + n_heads: 32 + n_kv_heads: 8 + vocab_size: 128256 + ffn_dim_multiplier: 1.3 + multiple_of: 1024 + norm_eps: 1e-05 + rope_theta: 500000.0 + weight_tying: false + max_seqlen: 6144 + freeze_language_model: false + freeze_vision_model: false + pooling_ratio: 2 + vision_model: + image_size: 448 + patch_size: 14 + width: 1536 + layers: 47 + heads: 16 + use_cls_token: false + use_abs_posemb: true + mlp_ratio: 5.833333334 + ls_init_value: 0.1 + drop_path: 0.1 + use_ln_post: false + pool_type: "none" + mlp_init: + use_gaussian: true + +data: + datamix: + num_workers: 4 + batch_size: 2 + image_res: 448 + max_num_tiles: 16 + max_video_frames: 16 + vision_input_type: thumb+tile + tokenizer_path: facebook/Perception-LM-8B/tokenizer.model + tokenizer_name: plmchat + conversation_format: plm_sft + +profiling: + run: false + +checkpoint: + dump: + every: 500 + keep: 1 + init_ckpt_path: + is_consolidated_model: True + +logging: + freq: 10 + level: INFO # Available choices for logging level are: [NOTSET, DEBUG, INFO, WARN, ERROR, FATAL, CRITICAL] diff --git a/perception_models/apps/plm/configs/stage_3/plm_1b.yaml b/perception_models/apps/plm/configs/stage_3/plm_1b.yaml new file mode 100644 index 0000000000000000000000000000000000000000..4a848e5f49f505f4d1707b84b713f582b8368093 --- /dev/null +++ b/perception_models/apps/plm/configs/stage_3/plm_1b.yaml @@ -0,0 +1,81 @@ +# We use a global batch size of 1024 in stage # 3 for PLM-1B model. Please adjust batch_size as per your training setup. +# For example, one possible configuration is batch_size=4,nodes=32,gpus_per_node=8 = 4*32*8 = 1024 global batch size. + +name: "plm_1b_stage3" +dump_dir: ./plm_1b_stage3 +steps: 21000 +seed: 777 +optim: + lr: 4e-5 + warmup: 120 + lr_min_ratio: 0.01 + clip: 1.0 + weight_decay: 0.01 + +distributed: + fsdp_type: full_shard + compile: false + model_dtype: bf16 + matmul_allow_tf32: false + selective_activation_checkpointing: false + full_activation_checkpointing: true + tp_size: 1 + +model: + dim: 2048 + n_layers: 16 + n_heads: 32 + n_kv_heads: 8 + vocab_size: 128256 + ffn_dim_multiplier: 1.5 + multiple_of: 256 + norm_eps: 1e-05 + rope_theta: 500000.0 + weight_tying: true + rope_scale_factor: 32 + high_freq_factor: 4 + max_seqlen: 11520 + freeze_language_model: false + freeze_vision_model: false + pooling_ratio: 2 + vision_model: + image_size: 448 + patch_size: 14 + width: 1024 + layers: 23 + heads: 16 + use_cls_token: true + use_abs_posemb: true + mlp_ratio: 4.0 + ls_init_value: 0.1 + drop_path: 0.1 + use_ln_post: false + pool_type: "none" + mlp_init: + use_gaussian: true + +data: + datamix: + num_workers: 4 + batch_size: 2 + image_res: 448 + max_num_tiles: 36 + max_video_frames: 32 + vision_input_type: thumb+tile + tokenizer_path: facebook/Perception-LM-1B/tokenizer.model + tokenizer_name: plmchat + conversation_format: plm_sft + +profiling: + run: false + +checkpoint: + dump: + every: 500 + keep: 1 + init_ckpt_path: + is_consolidated_model: True + +logging: + freq: 10 + level: INFO # Available choices for logging level are: [NOTSET, DEBUG, INFO, WARN, ERROR, FATAL, CRITICAL] diff --git a/perception_models/apps/plm/configs/stage_3/plm_3b.yaml b/perception_models/apps/plm/configs/stage_3/plm_3b.yaml new file mode 100644 index 0000000000000000000000000000000000000000..9ef19e80527d5a018f723189006c5b51a977e98a --- /dev/null +++ b/perception_models/apps/plm/configs/stage_3/plm_3b.yaml @@ -0,0 +1,81 @@ +# We use a global batch size of 1024 in stage # 3 for PLM-3B model. Please adjust batch_size as per your training setup. +# For example, one possible configuration is batch_size=4,nodes=32,gpus_per_node=8 = 4*32*8 = 1024 global batch size. + +name: "plm_3b_stage3" +dump_dir: ./plm_3b_stage3 +steps: 21000 +seed: 777 +optim: + lr: 4e-5 + warmup: 120 + lr_min_ratio: 0.01 + clip: 1.0 + weight_decay: 0.01 + +distributed: + fsdp_type: full_shard + compile: false + model_dtype: bf16 + matmul_allow_tf32: false + selective_activation_checkpointing: false + full_activation_checkpointing: true + tp_size: 1 + +model: + dim: 3072 + n_layers: 28 + n_heads: 24 + n_kv_heads: 8 + vocab_size: 128256 + ffn_dim_multiplier: 1.0 + multiple_of: 256 + norm_eps: 1e-05 + rope_theta: 500000.0 + weight_tying: true + rope_scale_factor: 32 + high_freq_factor: 4 + max_seqlen: 11520 + freeze_language_model: false + freeze_vision_model: false + pooling_ratio: 2 + vision_model: + image_size: 448 + patch_size: 14 + width: 1024 + layers: 23 + heads: 16 + use_cls_token: true + use_abs_posemb: true + mlp_ratio: 4.0 + ls_init_value: 0.1 + drop_path: 0.1 + use_ln_post: false + pool_type: "none" + mlp_init: + use_gaussian: true + +data: + datamix: + num_workers: 4 + batch_size: 2 + image_res: 448 + max_num_tiles: 36 + max_video_frames: 32 + vision_input_type: thumb+tile + tokenizer_path: facebook/Perception-LM-3B/tokenizer.model + tokenizer_name: plmchat + conversation_format: plm_sft + +profiling: + run: false + +checkpoint: + dump: + every: 500 + keep: 1 + init_ckpt_path: + is_consolidated_model: True + +logging: + freq: 10 + level: INFO # Available choices for logging level are: [NOTSET, DEBUG, INFO, WARN, ERROR, FATAL, CRITICAL] diff --git a/perception_models/apps/plm/configs/stage_3/plm_8b.yaml b/perception_models/apps/plm/configs/stage_3/plm_8b.yaml new file mode 100644 index 0000000000000000000000000000000000000000..2b7eb775a9ea3fa88343d34e5305600890640b87 --- /dev/null +++ b/perception_models/apps/plm/configs/stage_3/plm_8b.yaml @@ -0,0 +1,79 @@ +# We use a global batch size of 1024 in stage # 3 for PLM-8B model. Please adjust batch_size as per your training setup. +# For example, one possible configuration is batch_size=2,nodes=64,gpus_per_node=8 = 2*64*8 = 1024 global batch size. + +name: "plm_8b_stage3" +dump_dir: ./plm_8b_stage3 +steps: 21000 +seed: 777 +optim: + lr: 1e-5 + warmup: 120 + lr_min_ratio: 0.01 + clip: 1.0 + weight_decay: 0.05 + +distributed: + fsdp_type: full_shard + compile: false + model_dtype: bf16 + matmul_allow_tf32: false + selective_activation_checkpointing: false + full_activation_checkpointing: true + tp_size: 1 + +model: + dim: 4096 + n_layers: 32 + n_heads: 32 + n_kv_heads: 8 + vocab_size: 128256 + ffn_dim_multiplier: 1.3 + multiple_of: 1024 + norm_eps: 1e-05 + rope_theta: 500000.0 + weight_tying: false + max_seqlen: 11520 + freeze_language_model: false + freeze_vision_model: false + pooling_ratio: 2 + vision_model: + image_size: 448 + patch_size: 14 + width: 1536 + layers: 47 + heads: 16 + use_cls_token: false + use_abs_posemb: true + mlp_ratio: 5.833333334 + ls_init_value: 0.1 + drop_path: 0.1 + use_ln_post: false + pool_type: "none" + mlp_init: + use_gaussian: true + +data: + datamix: + num_workers: 4 + batch_size: 2 + image_res: 448 + max_num_tiles: 36 + max_video_frames: 32 + vision_input_type: thumb+tile + tokenizer_path: facebook/Perception-LM-8B/tokenizer.model + tokenizer_name: plmchat + conversation_format: plm_sft + +profiling: + run: false + +checkpoint: + dump: + every: 500 + keep: 1 + init_ckpt_path: + is_consolidated_model: True + +logging: + freq: 10 + level: INFO # Available choices for logging level are: [NOTSET, DEBUG, INFO, WARN, ERROR, FATAL, CRITICAL] diff --git a/perception_models/apps/plm/consolidate.py b/perception_models/apps/plm/consolidate.py new file mode 100644 index 0000000000000000000000000000000000000000..25d231b658937cc6ec012ecffea7b926c0b9eeea --- /dev/null +++ b/perception_models/apps/plm/consolidate.py @@ -0,0 +1,55 @@ +import argparse +import os +from pathlib import Path + +import torch +from omegaconf import OmegaConf + +from apps.plm.transformer import LMTransformer, LMTransformerArgs +from core.args import dataclass_from_dict +from core.checkpoint import load_from_checkpoint + + +def build_model( + ref_model_path: str, + model_cls=LMTransformer, + model_args_cls=LMTransformerArgs, +): + ckpt_path = Path(ref_model_path) + config = ckpt_path / "params.json" + config = OmegaConf.load(config) + + model_args = dataclass_from_dict(model_args_cls, config.model, strict=False) + model = model_cls(model_args) + return model + + +def main(): + parser = argparse.ArgumentParser(description="Consolidate PLM checkpoints") + parser.add_argument( + "--ckpt", + type=str, + required=True, + help="Path to the checkpoint directory to consolidate", + ) + args = parser.parse_args() + + model = build_model(ref_model_path=args.ckpt) + load_from_checkpoint( + ckpt_dir=args.ckpt, + model=model, + optimizer=None, + model_key="model", + ) + + consolidated_model_state_dict = model.state_dict() + output_file = os.path.join(args.ckpt, "consolidated.pth") + + # Save the consolidated model state_dict using torch.save + print(f"Saving consolidated model state_dict to: {output_file}") + torch.save(consolidated_model_state_dict, output_file) + print("Consolidated checkpoint saved successfully.") + + +if __name__ == "__main__": + main() diff --git a/perception_models/apps/plm/dataset_conf.py b/perception_models/apps/plm/dataset_conf.py new file mode 100644 index 0000000000000000000000000000000000000000..e6b3a5f22388b35022c613bf0cdc4a4e3e2b29de --- /dev/null +++ b/perception_models/apps/plm/dataset_conf.py @@ -0,0 +1,37 @@ +# Copyright (c) Meta Platforms, Inc. and affiliates. + +import os +from dataclasses import dataclass +from typing import Optional + +import yaml + + +@dataclass +class DatasetConf: + name: str = "" + annotation: str = "" + root_dir: Optional[str] = None + + +def read_yaml_to_configs(yaml_file_path: str) -> dict: + with open(yaml_file_path, "r", encoding="utf-8") as file: + yaml_data = yaml.safe_load(file) + + dataset_config = {} + for dataset_name, dataset_info in yaml_data.items(): + dataset_config[dataset_name] = DatasetConf( + name=dataset_name, + annotation=dataset_info["annotation"], + root_dir=dataset_info.get("root_dir"), + ) + + return dataset_config + + +# Determine the directory of the current script +current_directory = os.path.dirname(os.path.abspath(__file__)) +# Construct the path to the datasets.yaml file +yaml_file_path = os.path.join(current_directory, "configs", "datasets.yaml") +# Read the YAML file +dataset_config = read_yaml_to_configs(yaml_file_path) diff --git a/perception_models/apps/plm/docs/evaluation.md b/perception_models/apps/plm/docs/evaluation.md new file mode 100644 index 0000000000000000000000000000000000000000..6bd0af71c45f0b31ef28f16f3321feccecb9399e --- /dev/null +++ b/perception_models/apps/plm/docs/evaluation.md @@ -0,0 +1,48 @@ +# Evaluating Perception Language Model (PLM) + +[![Hugging Face Collection](https://img.shields.io/badge/%F0%9F%A4%97%20PLM 1B-Model-blue)](https://huggingface.co/facebook/Perception-LM-1B) +[![Hugging Face Collection](https://img.shields.io/badge/%F0%9F%A4%97%20PLM 3B-Model-blue)](https://huggingface.co/facebook/Perception-LM-3B) +[![Hugging Face Collection](https://img.shields.io/badge/%F0%9F%A4%97%20PLM 8B-Model-blue)](https://huggingface.co/facebook/Perception-LM-8B) + +We have added our model and benchmarks to [`lmms-eval`](https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/main/lmms_eval/models/simple/plm.py) for to support the process of reproducing our reported results on multiple image and video benchmarks. + +--- + +## Getting Started +1. Install perception_models following the instruction in the [`Main README`](../../../README.md). +2. Install `lmms-eval`: +``` +git clone https://github.com/EvolvingLMMs-Lab/lmms-eval.git +cd lmm-evals +pip install -e . +``` + +## Run Evaluation on Standard Image and Video Tasks +You can use the following command to run the evaluation. + +```shell + +# Use facebook/Perception-LM-1B for 1B parameters model and facebook/Perception-LM-8B for 8B parameters model. +CHECKPOINTS_PATH=facebook/Perception-LM-3B + +# Define the tasks you want to evaluate PLM on. We support all the tasks present in lmms-eval, however have tested the following tasks with our models. + +ALL_TASKS=( + "docvqa" "chartqa" "textvqa" "infovqa" "ai2d_no_mask" "ok_vqa" "vizwiz_vqa" "mme" + "realworldqa" "pope" "mmmu" "ocrbench" "coco_karpathy_val" "nocaps" "vqav2_val" + "mvbench" "videomme" "vatex_test" "egoschema" "egoschema_subset" "mlvu_dev" + "tempcompass_multi_choice" "perceptiontest_val_mc" "perceptiontest_test_mc" +) + +# After specifying the task/tasks to evaluate, run the following command to start the evaluation. +SELECTED_TASK="textvqa,videomme" +accelerate launch --num_processes=8 \ +-m lmms_eval \ +--model plm \ +--model_args pretrained=$CHECKPOINTS_PATH \ +--tasks $SELECTED_TASK \ +--batch_size 1 \ +--log_samples \ +--log_samples_suffix plm \ +--output_path $OUTPUT_PATH +``` diff --git a/perception_models/apps/plm/docs/finetune_example.md b/perception_models/apps/plm/docs/finetune_example.md new file mode 100644 index 0000000000000000000000000000000000000000..223a0421ed0165ba82bfc767ec3ff40efcb8d481 --- /dev/null +++ b/perception_models/apps/plm/docs/finetune_example.md @@ -0,0 +1,168 @@ +# Example to Finetune PLM on New Data + +We provide a step-by-step walkthrough for finetuning PLM on a custom dataset based on the high-level instructions in [training.md](training.md). For this example, we will finetune PLM-8B on a specific domain ([Radiology images](https://huggingface.co/datasets/unsloth/Radiology_mini)) and compare model performance before and after finetuning. + +### Setup +Install required packages: +```bash +pip install datasets tqdm +``` + + +### 1. Download dataset and prepare for training + +``` python +import json +import os +import tqdm +from datasets import load_dataset + +def convert_to_training_jsonl(dataset, split): + + out_dir = "apps/plm/dummy_datasets/Radiology_mini" + os.makedirs(f"{out_dir}/images", exist_ok=True) + + parsed_data = [] + for entry in tqdm.tqdm(dataset[split]): + + # save image + image_path = f"{out_dir}/images/{entry["image_id"]}.png" + entry["image"].save(image_path) + + # create training conversation template + conversations = [ + {"from": "human", "value": "You are an expert radiographer. Describe accurately what you see in this image."}, + {"from": "assistant", "value": entry["caption"]} + ] + + parsed_data.append({ + "image": f"{entry["image_id"]}.png", + "conversations": conversations, + }) + + # Write jsonl for training / evaluation + with open(f"{out_dir}/{split}.jsonl", "w") as f: + for entry in parsed_data: + f.write(json.dumps(entry) + "\n") + + +dataset = load_dataset("unsloth/Radiology_mini") +convert_to_training_jsonl(dataset, "train") +convert_to_training_jsonl(dataset, "test") +``` + +After running this code, the training data will be ready for use with the codebase: +``` +apps/plm/dummy_datasets/Radiology_mini +├── train.jsonl +├── test.jsonl +├── images +│ ├── ROCOv2_2023_test_000022.png +│ ├── ROCOv2_2023_train_059888.png +│ ├── ... +``` + +where each data jsonl will contain data in the required training format. +``` +# train.jsonl +{"image": "ROCOv2_2023_train_054311.png", "conversations": [{"from": "human", "value": "You are an expert radiographer. Describe accurately what you see in this image."}, {"from": "assistant", "value": "Panoramic radiography shows an osteolytic lesion in the right posterior maxilla with resorption of the floor of the maxillary sinus (arrows)."}]} +{"image": "ROCOv2_2023_train_058916.png", "conversations": [{"from": "human", "value": "You are an expert radiographer. Describe accurately what you see in this image."}, {"from": "assistant", "value": "ERCP showing distal CBD compression. ERCP - endoscopic retrograde cholangiopancreatography; CBD - common bile duct"}]} +... +``` + + +### 2. Add dataset config to configs/datasets.yaml +Point to the newly created data in [configs/datasets.yaml](../configs/datasets.yaml) by adding these lines at the bottom. +``` +radiology_finetune: + annotation: apps/plm/dummy_datasets/Radiology_mini/train.jsonl + root_dir: apps/plm/dummy_datasets/Radiology_mini/images +``` + +### 3. Copy and modify the provided finetuning config +The stage # 3 configs can be used to further finetune PLM [configs/stage_3](../configs/stage_3). +```bash +cp apps/plm/configs/stage_3/plm_8b.yaml apps/plm/configs/finetune/plm_8b_custom.yaml +``` + +Copy the config and modify the fields below. +```yaml +# Set the path to save checkpoints to +dump_dir: checkpoints/finetune_example/ + +# Total number of training iterations +steps: 500 + +# Pointer to previously created datamix. Ideally, you would incorporate the new data into a larger datamix +# but for now, we finetune only on this data +data: + datamix: radiology_finetune:1 + +# Pointer to the initial model weights +checkpoint: + init_ckpt_path: facebook/Perception-LM-8B +``` + +Various other parameters can be changed such as learning rate, batch_size, etc. See comments in [configs/stage_3/plm_8b.yaml](../configs/stage_3/plm_8b.yaml) for details. + +### 4. Finetune the model +Finetune a model on a single node. For multi-node training, refer to the main [training.md](training.md) doc. +``` +torchrun --nproc-per-node 8 -m apps.plm.train \ + config=apps/plm/configs/finetune/plm_8b_custom.yaml +``` + +This will start training and save checkpoints, logs and configs in the previously specified `dump_dir`. +``` +checkpoints/finetune_example/ +├── checkpoints +│ └── 0000000500 +│ ├── __0_0.distcp +│ ├── __1_0.distcp +│ ├── ... +│ ├── params.json +│ ├── train_state_00000.json +│ ├── train_state_00001.json +│ ├── ... +├── config.yaml +├── metrics.jsonl +└── train.log +``` + +### 5. Consolidate the checkpoint +Models trained with FSDP require their weights to be consolidated before inference to create `consolidated.pth`. +```bash +python apps/plm/consolidate.py --ckpt checkpoints/finetune_example/checkpoints/0000000500/ +``` + +### 6. Test and compare model generation +Use the provided generate helper script to compare the base model (before finetuning) to the finetuned version on an unseen test image from the same dataset. + +```bash +python apps/plm/generate.py \ + --ckpt facebook/Perception-LM-8B \ + --media_type image \ + --media_path apps/plm/dummy_datasets/Radiology_mini/images/ROCOv2_2023_test_000022.png \ + --question 'You are an expert radiographer. Describe accurately what you see in this image.' + +# Generation: +# The image is a medical scan of a person's abdomen, likely an MRI or CT scan. The scan shows the internal organs of the abdomen, including the liver, stomach, and intestines. The liver is located on the left side of the image, and it appears to be slightly enlarged. The stomach is located in the center of the image, and it appears to be normal in size. The intestines are located on the right side of the image, and they appear to be normal in size and shape. There are no visible abnormalities or tumors in the image. The scan is in black and white, with the organs appearing in shades of gray. The background of the image is black, which helps to highlight the details of the organs. Overall, the image suggests that the person's abdominal organs are healthy and normal. +``` + + +```bash +python apps/plm/generate.py \ + --ckpt checkpoints/finetune_example/checkpoints/0000000500/ \ + --media_type image \ + --media_path apps/plm/dummy_datasets/Radiology_mini/images/ROCOv2_2023_test_000022.png \ + --question 'You are an expert radiographer. Describe accurately what you see in this image.' + +# Generation: +# CT scan of the abdomen demonstrating a large liver metastasis (yellow arrow) in segment VII. +``` + +Comparing the two, we see the finetuned model provide concise descriptions following the style of the training set. Note that we use the same prompt as training since the dataset is small and the model has likely overfit to it. For robust training, include the new data in a large data mix (e.g., our provided [SFT blend](../configs/stage_3/plm_8b.yaml)). + + +### Wrap up +From here, the model is trained and ready for evaluation. The [generation script](../generate.py) can be modified to directly evaluate the model on the radiology image captioning task (test set) using captioning metrics (e.g., CIDEr). Alternately, if trained with a larger SFT blend, it can be used for domain-specific QA (e.g., [VQA-Radiology](https://huggingface.co/datasets/flaviagiammarino/vqa-rad)). \ No newline at end of file diff --git a/perception_models/apps/plm/docs/plm_main_fig.png b/perception_models/apps/plm/docs/plm_main_fig.png new file mode 100644 index 0000000000000000000000000000000000000000..c237268b2ae0edb1ac10f2541e3081c990c3d8d4 --- /dev/null +++ b/perception_models/apps/plm/docs/plm_main_fig.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:309f14e6c7bcaa3d2d4b8bffea1030f276224398bf976e507efc839ab43c53de +size 544743 diff --git a/perception_models/apps/plm/docs/plm_videobench.md b/perception_models/apps/plm/docs/plm_videobench.md new file mode 100644 index 0000000000000000000000000000000000000000..93016b0bdf1217a85c3ad02e8d5d18b70cfa6355 --- /dev/null +++ b/perception_models/apps/plm/docs/plm_videobench.md @@ -0,0 +1,54 @@ +# PLM-VideoBench +[![Hugging Face Collection](https://img.shields.io/badge/%F0%9F%A4%97%20PLM‑VideoBench-BenchMark-blue)](https://huggingface.co/datasets/facebook/PLM-VideoBench) + +As part of our PLM-release, we are releasing a comprehensive set of video benchmarks (grouped as `PLM-VideoBench`) for detailed video understanding. PLM-VideoBench includes the following sub-benchmarks, +1. **Fine-Grained Question Answering (FGQA):** In this task, a model must answer a multiple-choice question (MCQ) +that probes fine-grained activity understanding. +2. **Smart Glasses Question Answering (SGQA):** In this task, a model must answer open-ended questions about +activities and objects visible in an egocentric video stream recorded by a Meta VR Glasses. +3. **Video Region Captioning (RCap):** In this task, the model must generate a detailed description of an event +involving a subject of interest in the video. +4. **Region Temporal Localization (RTLoc):** In this task, the model must identify the precise time interval within the video when the specified event takes place for the given subject. +5. **Region Dense Video Captioning (RDCap):** In this task, a model must generate a detailed description of all events involving a specific subject of interest in a video. + +> [!TIP] +> We have added all `PLM-VideoBench` tasks to [`lmms-eval`](https://github.com/EvolvingLMMs-Lab/lmms-eval/tree/main/lmms_eval/tasks/plm_videobench). This makes it easy to reproduce PLM results and also allows other models to be tested on the benchmarks. + +You can use the following command to evaluate PLM on PLM-VideoBench. + +```shell + +# Use facebook/Perception-LM-1B for 1B parameters model and facebook/Perception-LM-8B for 8B parameters model. +CHECKPOINTS_PATH=facebook/Perception-LM-3B. + +# PLM-VideoBench Tasks +SELECTED_TASK=fgqa_test,sgqa_test,rtloc_test,rcap_test,rdcap_test +OUTPUT_PATH="plm_videobench_evaluation" + +accelerate launch --num_processes=8 \ +-m lmms_eval \ +--model plm \ +--model_args pretrained=$CHECKPOINTS_PATH \ +--tasks $TASKS \ +--batch_size 1 \ +--log_samples \ +--log_samples_suffix plm \ +--output_path $OUTPUT_PATH +``` + +## Results + +We evaluate PLM against baselines on PLM-VideoBench and +report breakdowns. We report human performance in the first row. +| Model | FGQA (MBacc) | SGQA (Acc) | RDCap (SODA) | RCap (Score) | RTLoc (meanR) | Avg. | +|------------------|------|------|------------|------------|-------------|------| +| Human perf. | 90.9 | 67.9 | 66.6 | 53.9 | 67.8 | 73.9 | +| GPT-4o | 61.2 | **63.7** | 20.9 | 35.7 | 33.1 | 51.6 | +| Gemini 1.5 Pro | 57.1 | 49.9 | 14.4 | 33.1 | 27.6 | 44.0 | +| Gemini 2.0 Flash | 58.7 | 44.8 | 13.2 | 30.9 | 27.6 | 42.5 | +| LLaVA-OV-7B | 40.2 | 41.5 | 4.7 | 24.4 | 13.9 | 32.0 | +| Qwen2VL-7B | 49.2 | 44.5 | 4.1 | 17.6 | 15.1 | 35.3 | +| Qwen2.5VL-7B | 49.8 | 43.0 | 2.5 | 21.5 | 10.7 | 34.8 | +| InternVL2-8B | 47.7 | 45.9 | 1.2 | 21.5 | 11.6 | 35.0 | +| InternVL2.5-8B | 53.7 | 48.3 | 5.7 | 26.1 | 8.8 | 38.5 | +| PLM-8B | **67.7** | 46.2 | **52.8** | **46.6** | **59.1** | **55.6** | diff --git a/perception_models/apps/plm/docs/training.md b/perception_models/apps/plm/docs/training.md new file mode 100644 index 0000000000000000000000000000000000000000..abe235e764aeb7864cc361b0d0d9906d5cfe8557 --- /dev/null +++ b/perception_models/apps/plm/docs/training.md @@ -0,0 +1,113 @@ +# Training Perception Language Model (PLM) + +[![Hugging Face Collection](https://img.shields.io/badge/%F0%9F%A4%97%20PLM Synthetic-Image-blue)](https://huggingface.co/datasets/facebook/PLM-Image-Auto) +[![Hugging Face Collection](https://img.shields.io/badge/%F0%9F%A4%97%20PLM Synthetic-Video-blue)](https://huggingface.co/datasets/facebook/PLM-Video-Auto) +[![Hugging Face Collection](https://img.shields.io/badge/%F0%9F%A4%97%20PLM Human-Video-blue)](https://huggingface.co/datasets/facebook/PLM-Video-Human) + +We provide instruction to train or finetune PLM on a custom dataset. + +--- + +> [!TIP] +> We provide configurations to run [`warm-up`](../configs/warmup/) and [`sft`](../configs/sft/) to facilitate reproducibility of PLM training. + + +## Data Format :open_file_folder: + +We use support both image and video conversation datasets using `jsonl`. Each line of `jsonl` file should follow the following format, + +### For Image Conversation Dataset +```json + { + "image": "", + "conversations": [ + { + "from": "human", + "value": "human instruction" + }, + { + "from": "assistant", + "value": "model response" + } + ] + } +``` + +### For Video Conversation Dataset +```json + { + "video": "