Spaces:
Running
Running
| <html> | |
| <head> | |
| <meta charset="utf-8"> | |
| <!-- Meta tags for social media banners, these should be filled in appropriatly as they are your "business card" --> | |
| <!-- Replace the content tag with appropriate information --> | |
| <meta name="description" content="DESCRIPTION META TAG"> | |
| <meta property="og:title" content="SOCIAL MEDIA TITLE TAG"/> | |
| <meta property="og:description" content="SOCIAL MEDIA DESCRIPTION TAG TAG"/> | |
| <meta property="og:url" content="URL OF THE WEBSITE"/> | |
| <!-- Path to banner image, should be in the path listed below. Optimal dimenssions are 1200X630--> | |
| <meta property="og:image" content="static/image/your_banner_image.png" /> | |
| <meta property="og:image:width" content="1200"/> | |
| <meta property="og:image:height" content="630"/> | |
| <meta name="twitter:title" content="TWITTER BANNER TITLE META TAG"> | |
| <meta name="twitter:description" content="TWITTER BANNER DESCRIPTION META TAG"> | |
| <!-- Path to banner image, should be in the path listed below. Optimal dimenssions are 1200X600--> | |
| <meta name="twitter:image" content="static/images/your_twitter_banner_image.png"> | |
| <meta name="twitter:card" content="summary_large_image"> | |
| <!-- Keywords for your paper to be indexed by--> | |
| <meta name="keywords" content="KEYWORDS SHOULD BE PLACED HERE"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1"> | |
| <title>Pano2IR</title> | |
| <!-- <link rel="icon" type="image/x-icon" href="static/images/favicon.ico"> --> | |
| <link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" | |
| rel="stylesheet"> | |
| <link rel="stylesheet" href="static/css/bulma.min.css"> | |
| <link rel="stylesheet" href="static/css/bulma-carousel.min.css"> | |
| <link rel="stylesheet" href="static/css/bulma-slider.min.css"> | |
| <link rel="stylesheet" href="static/css/fontawesome.all.min.css"> | |
| <link rel="stylesheet" | |
| href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css"> | |
| <link rel="stylesheet" href="static/css/index.css"> | |
| <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script> | |
| <script src="https://documentcloud.adobe.com/view-sdk/main.js"></script> | |
| <script defer src="static/js/fontawesome.all.min.js"></script> | |
| <script src="static/js/bulma-carousel.min.js"></script> | |
| <script src="static/js/bulma-slider.min.js"></script> | |
| <script src="static/js/index.js"></script> | |
| <script id="MathJax-script" async | |
| src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"> | |
| </script> | |
| </head> | |
| <body> | |
| <section class="hero"> | |
| <div class="hero-body"> | |
| <div class="container is-max-desktop"> | |
| <div class="columns is-centered"> | |
| <div class="column has-text-centered"> | |
| <h1 class="title is-1 publication-title">Pano2IR: Spatial Room Impulse Responses Generation <br> from Egocentric Panoramic Image</h1> | |
| <div class="is-size-4 publication-authors"> | |
| <!-- Paper authors --> | |
| <span class="author-block"> | |
| Anonymous Authors | |
| </span> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| </section> | |
| <!-- Single image with description and fixed width --> | |
| <section class="hero is-small"> | |
| <div class="hero-body"> | |
| <div class="container"> | |
| <figure class="image" style="margin: 0 auto; max-width: 1280px;"> | |
| <img src="static/images/statement.png" alt="Result Image" style="width: 100%; height: auto;"> | |
| </figure> | |
| <div class="container is-max-desktop mt-6"> | |
| <div class="columns is-centered"> | |
| <div class="column is-three-quarter"> | |
| <h2 class="title is-2 has-text-centered mb-5">Abstract</h2> | |
| <div class="content has-text-justified" style="font-size: 1.2rem" > | |
| <p> | |
| Immersive spatial audio for augmented and extended reality (AR/XR) applications requires alignment with visual cues from the surrounding environment. However, inferring realistic room impulse responses (RIRs) from visual information in diverse and unconstrained environments remains a major challenge. <font color="#497c27"><b>We propose Pano2IR, a framework that generates first-order Ambisonics (FOA) spatial room impulse responses (SRIRs) from a single 360° egocentric panoramic image captured at the user's location.</b></font> This enables user-centric spatial audio rendering in arbitrary environments without the need for extensive RIR measurements or full 3D meshes. To preserve the spatial characteristics of SRIRs, we introduce a diffuseness-weighted intensity vector (Diff-IV) loss, which captures inter-channel relationships by leveraging both the intensity vector and diffuseness inherent in FOA-SRIRs. Pano2IR extracts three acoustic-correlated visual transformations from a single RGB panorama—depth maps, room geometry, and object absorption maps—to effectively bridge the visual-acoustic domain gap. In addition, we encode key acoustic properties of SRIRs into latent acoustic parameter representations. A contrastive learning scheme then aligns these acoustic features with their corresponding visual counterparts, enabling the model to distinguish matching acoustic-visual pairs from non-matching ones. | |
| </p> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| </section> | |
| <!-- Experimental Results --> | |
| <section class="hero is-small is-light"> | |
| <div class="hero-body"> | |
| <div class="container is-max-desktop mt-6"> | |
| <div class="columns is-centered"> | |
| <div class="column is-three-quarter"> | |
| <h2 class="title is-2 has-text-centered mb-5">Experimental Results</h2> | |
| <div class="content has-text-justified" style="font-size: 1.2rem"> | |
| <p> | |
| The effectiveness of Pano2IR is showcased by inference results on panoramic images captured in <font color="#da2424"><b>previously unseen scenes</b></font>. The generated FOA-SRIRs were transformed into binaural audio using the off-the-shelf ambisonics-to-binaural plug-in. | |
| </p> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| </section> | |
| <!-- Results Section --> | |
| <section class="section"> | |
| <div class="container" style="max-width:1280px; margin:0 auto;"> | |
| <div class="columns is-multiline"> | |
| <!-- 1번째 샘플 --> | |
| <div class="column is-full"> | |
| <div class="columns is-vcentered" style="align-items: stretch;"> | |
| <div class="column is-half is-flex" style="display: flex; align-items: stretch;"> | |
| <div class="card mb-5" style="width: 100%; height: 100%; display: flex; flex-direction: column; justify-content: center;"> | |
| <div class="card-content has-text-centered" style="flex: 1 1 auto; display: flex; align-items: center; justify-content: center;"> | |
| <figure class="image" style="width: 100%; margin: 0;"> | |
| <img src="static/results/1_5ZKStnWn8Zo_43/0_test_MP3D_ACN_N3D_5ZKStnWn8Zo_43-ir_0img.png" alt="Panorama 1" style="width: 100%; aspect-ratio: 2 / 1; object-fit: cover;"> | |
| </figure> | |
| </div> | |
| </div> | |
| </div> | |
| <div class="column is-half is-flex" style="display: flex; align-items: stretch;"> | |
| <div class="card mb-5" style="width: 100%; height: 100%; display: flex; flex-direction: column; justify-content: center;"> | |
| <div class="card-content" style="flex: 1 1 auto; display: flex; flex-direction: column; justify-content: center;"> | |
| <div class="mb-3" style="font-size: 1.3rem; font-weight: bold;">Original audio</div> | |
| <audio controls style="width: 100%;"> | |
| <source src="static/results/1_5ZKStnWn8Zo_43/48k_0_test_MP3D_ACN_N3D_5ZKStnWn8Zo_43_jembe_orig.wav" type="audio/wav"> | |
| 브라우저가 오디오 태그를 지원하지 않습니다. | |
| </audio> | |
| <div class="mt-4 mb-3" style="font-size: 1.3rem; font-weight: bold;">Generated audio</div> | |
| <audio controls style="width: 100%;"> | |
| <source src="static/results/1_5ZKStnWn8Zo_43/48k_0_test_MP3D_ACN_N3D_5ZKStnWn8Zo_43_jembe_gen.wav" type="audio/wav"> | |
| 브라우저가 오디오 태그를 지원하지 않습니다. | |
| </audio> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| <!-- 2번째 샘플 --> | |
| <div class="column is-full"> | |
| <div class="columns is-vcentered" style="align-items: stretch;"> | |
| <div class="column is-half is-flex" style="display: flex; align-items: stretch;"> | |
| <div class="card mb-5" style="width: 100%; height: 100%; display: flex; flex-direction: column; justify-content: center;"> | |
| <div class="card-content has-text-centered" style="flex: 1 1 auto; display: flex; align-items: center; justify-content: center;"> | |
| <figure class="image" style="width: 100%; margin: 0;"> | |
| <img src="static/results/2_UwV83HsGsw3_740/0_test_MP3D_ACN_N3D_UwV83HsGsw3_740-ir_0img.png" alt="Panorama 2" style="width: 100%; aspect-ratio: 2 / 1; object-fit: cover;"> | |
| </figure> | |
| </div> | |
| </div> | |
| </div> | |
| <div class="column is-half is-flex" style="display: flex; align-items: stretch;"> | |
| <div class="card mb-5" style="width: 100%; height: 100%; display: flex; flex-direction: column; justify-content: center;"> | |
| <div class="card-content" style="flex: 1 1 auto; display: flex; flex-direction: column; justify-content: center;"> | |
| <div class="mb-3" style="font-size: 1.3rem; font-weight: bold;">Original audio</div> | |
| <audio controls style="width: 100%;"> | |
| <source src="static/results/2_UwV83HsGsw3_740/48k_0_test_MP3D_ACN_N3D_UwV83HsGsw3_740_jembe_orig.wav" type="audio/wav"> | |
| 브라우저가 오디오 태그를 지원하지 않습니다. | |
| </audio> | |
| <div class="mt-4 mb-3" style="font-size: 1.3rem; font-weight: bold;">Generated audio</div> | |
| <audio controls style="width: 100%;"> | |
| <source src="static/results/2_UwV83HsGsw3_740/48k_0_test_MP3D_ACN_N3D_UwV83HsGsw3_740_jembe_gen.wav" type="audio/wav"> | |
| 브라우저가 오디오 태그를 지원하지 않습니다. | |
| </audio> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| <!-- 3번째 샘플 --> | |
| <div class="column is-full"> | |
| <div class="columns is-vcentered" style="align-items: stretch;"> | |
| <div class="column is-half is-flex" style="display: flex; align-items: stretch;"> | |
| <div class="card mb-5" style="width: 100%; height: 100%; display: flex; flex-direction: column; justify-content: center;"> | |
| <div class="card-content has-text-centered" style="flex: 1 1 auto; display: flex; align-items: center; justify-content: center;"> | |
| <figure class="image" style="width: 100%; margin: 0;"> | |
| <img src="static/results/3_rqfALeAoiTq_1835/0_test_MP3D_ACN_N3D_rqfALeAoiTq_1835-ir_0img.png" alt="Panorama 3" style="width: 100%; aspect-ratio: 2 / 1; object-fit: cover;"> | |
| </figure> | |
| </div> | |
| </div> | |
| </div> | |
| <div class="column is-half is-flex" style="display: flex; align-items: stretch;"> | |
| <div class="card mb-5" style="width: 100%; height: 100%; display: flex; flex-direction: column; justify-content: center;"> | |
| <div class="card-content" style="flex: 1 1 auto; display: flex; flex-direction: column; justify-content: center;"> | |
| <div class="mb-3" style="font-size: 1.3rem; font-weight: bold;">Original audio</div> | |
| <audio controls style="width: 100%;"> | |
| <source src="static/results/3_rqfALeAoiTq_1835/48k_0_test_MP3D_ACN_N3D_rqfALeAoiTq_1835_jembe_orig.wav" type="audio/wav"> | |
| 브라우저가 오디오 태그를 지원하지 않습니다. | |
| </audio> | |
| <div class="mt-4 mb-3" style="font-size: 1.3rem; font-weight: bold;">Generated audio</div> | |
| <audio controls style="width: 100%;"> | |
| <source src="static/results/3_rqfALeAoiTq_1835/48k_0_test_MP3D_ACN_N3D_rqfALeAoiTq_1835_jembe_gen.wav" type="audio/wav"> | |
| 브라우저가 오디오 태그를 지원하지 않습니다. | |
| </audio> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| <!-- 5번째 샘플 --> | |
| <div class="column is-full"> | |
| <div class="columns is-vcentered" style="align-items: stretch;"> | |
| <div class="column is-half is-flex" style="display: flex; align-items: stretch;"> | |
| <div class="card mb-5" style="width: 100%; height: 100%; display: flex; flex-direction: column; justify-content: center;"> | |
| <div class="card-content has-text-centered" style="flex: 1 1 auto; display: flex; align-items: center; justify-content: center;"> | |
| <figure class="image" style="width: 100%; margin: 0;"> | |
| <img src="static/results/5_5ZKStnWn8Zo_370/0_test_MP3D_ACN_N3D_5ZKStnWn8Zo_370-ir_0img.png" alt="Panorama 5" style="width: 100%; aspect-ratio: 2 / 1; object-fit: cover;"> | |
| </figure> | |
| </div> | |
| </div> | |
| </div> | |
| <div class="column is-half is-flex" style="display: flex; align-items: stretch;"> | |
| <div class="card mb-5" style="width: 100%; height: 100%; display: flex; flex-direction: column; justify-content: center;"> | |
| <div class="card-content" style="flex: 1 1 auto; display: flex; flex-direction: column; justify-content: center;"> | |
| <div class="mb-3" style="font-size: 1.3rem; font-weight: bold;">Original audio</div> | |
| <audio controls style="width: 100%;"> | |
| <source src="static/results/5_5ZKStnWn8Zo_370/48k_0_test_MP3D_ACN_N3D_5ZKStnWn8Zo_370_jembe_orig.wav" type="audio/wav"> | |
| 브라우저가 오디오 태그를 지원하지 않습니다. | |
| </audio> | |
| <div class="mt-4 mb-3" style="font-size: 1.3rem; font-weight: bold;">Generated audio</div> | |
| <audio controls style="width: 100%;"> | |
| <source src="static/results/5_5ZKStnWn8Zo_370/48k_0_test_MP3D_ACN_N3D_5ZKStnWn8Zo_370_jembe_gen.wav" type="audio/wav"> | |
| 브라우저가 오디오 태그를 지원하지 않습니다. | |
| </audio> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| </section> | |