-
LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMs
Authors:
LLM-jp,
:,
Akiko Aizawa,
Eiji Aramaki,
Bowen Chen,
Fei Cheng,
Hiroyuki Deguchi,
Rintaro Enomoto,
Kazuki Fujii,
Kensuke Fukumoto,
Takuya Fukushima,
Namgi Han,
Yuto Harada,
Chikara Hashimoto,
Tatsuya Hiraoka,
Shohei Hisada,
Sosuke Hosokawa,
Lu Jie,
Keisuke Kamata,
Teruhito Kanazawa,
Hiroki Kanezashi,
Hiroshi Kataoka,
Satoru Katsumata,
Daisuke Kawahara,
Seiya Kawano
, et al. (57 additional authors not shown)
Abstract:
This paper introduces LLM-jp, a cross-organizational project for the research and development of Japanese large language models (LLMs). LLM-jp aims to develop open-source and strong Japanese LLMs, and as of this writing, more than 1,500 participants from academia and industry are working together for this purpose. This paper presents the background of the establishment of LLM-jp, summaries of its…
▽ More
This paper introduces LLM-jp, a cross-organizational project for the research and development of Japanese large language models (LLMs). LLM-jp aims to develop open-source and strong Japanese LLMs, and as of this writing, more than 1,500 participants from academia and industry are working together for this purpose. This paper presents the background of the establishment of LLM-jp, summaries of its activities, and technical reports on the LLMs developed by LLM-jp. For the latest activities, visit https://llm-jp.nii.ac.jp/en/.
△ Less
Submitted 4 July, 2024;
originally announced July 2024.
-
E2GS: Event Enhanced Gaussian Splatting
Authors:
Hiroyuki Deguchi,
Mana Masuda,
Takuya Nakabayashi,
Hideo Saito
Abstract:
Event cameras, known for their high dynamic range, absence of motion blur, and low energy usage, have recently found a wide range of applications thanks to these attributes. In the past few years, the field of event-based 3D reconstruction saw remarkable progress, with the Neural Radiance Field (NeRF) based approach demonstrating photorealistic view synthesis results. However, the volume rendering…
▽ More
Event cameras, known for their high dynamic range, absence of motion blur, and low energy usage, have recently found a wide range of applications thanks to these attributes. In the past few years, the field of event-based 3D reconstruction saw remarkable progress, with the Neural Radiance Field (NeRF) based approach demonstrating photorealistic view synthesis results. However, the volume rendering paradigm of NeRF necessitates extensive training and rendering times. In this paper, we introduce Event Enhanced Gaussian Splatting (E2GS), a novel method that incorporates event data into Gaussian Splatting, which has recently made significant advances in the field of novel view synthesis. Our E2GS effectively utilizes both blurry images and event data, significantly improving image deblurring and producing high-quality novel view synthesis. Our comprehensive experiments on both synthetic and real-world datasets demonstrate our E2GS can generate visually appealing renderings while offering faster training and rendering speed (140 FPS). Our code is available at https://github.com/deguchihiroyuki/E2GS.
△ Less
Submitted 21 June, 2024;
originally announced June 2024.
-
Online Embedding Multi-Scale CLIP Features into 3D Maps
Authors:
Shun Taguchi,
Hideki Deguchi
Abstract:
This study introduces a novel approach to online embedding of multi-scale CLIP (Contrastive Language-Image Pre-Training) features into 3D maps. By harnessing CLIP, this methodology surpasses the constraints of conventional vocabulary-limited methods and enables the incorporation of semantic information into the resultant maps. While recent approaches have explored the embedding of multi-modal feat…
▽ More
This study introduces a novel approach to online embedding of multi-scale CLIP (Contrastive Language-Image Pre-Training) features into 3D maps. By harnessing CLIP, this methodology surpasses the constraints of conventional vocabulary-limited methods and enables the incorporation of semantic information into the resultant maps. While recent approaches have explored the embedding of multi-modal features in maps, they often impose significant computational costs, lacking practicality for exploring unfamiliar environments in real time. Our approach tackles these challenges by efficiently computing and embedding multi-scale CLIP features, thereby facilitating the exploration of unfamiliar environments through real-time map generation. Moreover, the embedding CLIP features into the resultant maps makes offline retrieval via linguistic queries feasible. In essence, our approach simultaneously achieves real-time object search and mapping of unfamiliar environments. Additionally, we propose a zero-shot object-goal navigation system based on our mapping approach, and we validate its efficacy through object-goal navigation, offline object retrieval, and multi-object-goal navigation in both simulated environments and real robot experiments. The findings demonstrate that our method not only exhibits swifter performance than state-of-the-art mapping methods but also surpasses them in terms of the success rate of object-goal navigation tasks.
△ Less
Submitted 26 March, 2024;
originally announced March 2024.
-
Language to Map: Topological map generation from natural language path instructions
Authors:
Hideki Deguchi,
Kazuki Shibata,
Shun Taguchi
Abstract:
In this paper, a method for generating a map from path information described using natural language (textual path) is proposed. In recent years, robotics research mainly focus on vision-and-language navigation (VLN), a navigation task based on images and textual paths. Although VLN is expected to facilitate user instructions to robots, its current implementation requires users to explain the detai…
▽ More
In this paper, a method for generating a map from path information described using natural language (textual path) is proposed. In recent years, robotics research mainly focus on vision-and-language navigation (VLN), a navigation task based on images and textual paths. Although VLN is expected to facilitate user instructions to robots, its current implementation requires users to explain the details of the path for each navigation session, which results in high explanation costs for users. To solve this problem, we proposed a method that creates a map as a topological map from a textual path and automatically creates a new path using this map. We believe that large language models (LLMs) can be used to understand textual path. Therefore, we propose and evaluate two methods, one for storing implicit maps in LLMs, and the other for generating explicit maps using LLMs. The implicit map is in the LLM's memory. It is created using prompts. In the explicit map, a topological map composed of nodes and edges is constructed and the actions at each node are stored. This makes it possible to estimate the path and actions at waypoints on an undescribed path, if enough information is available. Experimental results on path instructions generated in a real environment demonstrate that generating explicit maps achieves significantly higher accuracy than storing implicit maps in the LLMs.
△ Less
Submitted 15 March, 2024;
originally announced March 2024.
-
Centroid-Based Efficient Minimum Bayes Risk Decoding
Authors:
Hiroyuki Deguchi,
Yusuke Sakai,
Hidetaka Kamigaito,
Taro Watanabe,
Hideki Tanaka,
Masao Utiyama
Abstract:
Minimum Bayes risk (MBR) decoding achieved state-of-the-art translation performance by using COMET, a neural metric that has a high correlation with human evaluation. However, MBR decoding requires quadratic time since it computes the expected score between a translation hypothesis and all reference translations. We propose centroid-based MBR (CBMBR) decoding to improve the speed of MBR decoding.…
▽ More
Minimum Bayes risk (MBR) decoding achieved state-of-the-art translation performance by using COMET, a neural metric that has a high correlation with human evaluation. However, MBR decoding requires quadratic time since it computes the expected score between a translation hypothesis and all reference translations. We propose centroid-based MBR (CBMBR) decoding to improve the speed of MBR decoding. Our method clusters the reference translations in the feature space, and then calculates the score using the centroids of each cluster. The experimental results show that our CBMBR not only improved the decoding speed of the expected score calculation 5.7 times, but also outperformed vanilla MBR decoding in translation quality by up to 0.5 COMET in the WMT'22 En$\leftrightarrow$Ja, En$\leftrightarrow$De, En$\leftrightarrow$Zh, and WMT'23 En$\leftrightarrow$Ja translation tasks.
△ Less
Submitted 11 June, 2024; v1 submitted 17 February, 2024;
originally announced February 2024.
-
CLIP feature-based randomized control using images and text for multiple tasks and robots
Authors:
Kazuki Shibata,
Hideki Deguchi,
Shun Taguchi
Abstract:
This study presents a control framework leveraging vision language models (VLMs) for multiple tasks and robots. Notably, existing control methods using VLMs have achieved high performance in various tasks and robots in the training environment. However, these methods incur high costs for learning control policies for tasks and robots other than those in the training environment. Considering the ap…
▽ More
This study presents a control framework leveraging vision language models (VLMs) for multiple tasks and robots. Notably, existing control methods using VLMs have achieved high performance in various tasks and robots in the training environment. However, these methods incur high costs for learning control policies for tasks and robots other than those in the training environment. Considering the application of industrial and household robots, learning in novel environments where robots are introduced is challenging. To address this issue, we propose a control framework that does not require learning control policies. Our framework combines the vision-language CLIP model with a randomized control. CLIP computes the similarity between images and texts by embedding them in the feature space. This study employs CLIP to compute the similarity between camera images and text representing the target state. In our method, the robot is controlled by a randomized controller that simultaneously explores and increases the similarity gradients. Moreover, we fine-tune the CLIP to improve the performance of the proposed method. Consequently, we confirm the effectiveness of our approach through a multitask simulation and a real robot experiment using a two-wheeled robot and robot arm.
△ Less
Submitted 18 January, 2024;
originally announced January 2024.
-
knn-seq: Efficient, Extensible kNN-MT Framework
Authors:
Hiroyuki Deguchi,
Hayate Hirano,
Tomoki Hoshino,
Yuto Nishida,
Justin Vasselli,
Taro Watanabe
Abstract:
k-nearest-neighbor machine translation (kNN-MT) boosts the translation quality of a pre-trained neural machine translation (NMT) model by utilizing translation examples during decoding. Translation examples are stored in a vector database, called a datastore, which contains one entry for each target token from the parallel data it is made from. Due to its size, it is computationally expensive both…
▽ More
k-nearest-neighbor machine translation (kNN-MT) boosts the translation quality of a pre-trained neural machine translation (NMT) model by utilizing translation examples during decoding. Translation examples are stored in a vector database, called a datastore, which contains one entry for each target token from the parallel data it is made from. Due to its size, it is computationally expensive both to construct and to retrieve examples from the datastore. In this paper, we present an efficient and extensible kNN-MT framework, knn-seq, for researchers and developers that is carefully designed to run efficiently, even with a billion-scale large datastore. knn-seq is developed as a plug-in on fairseq and easy to switch models and kNN indexes. Experimental results show that our implemented kNN-MT achieves a comparable gain to the original kNN-MT, and the billion-scale datastore construction took 2.21 hours in the WMT'19 German-to-English translation task. We publish our knn-seq as an MIT-licensed open-source project and the code is available on https://github.com/naist-nlp/knn-seq . The demo video is available on https://youtu.be/zTDzEOq80m0 .
△ Less
Submitted 18 October, 2023;
originally announced October 2023.
-
Agent-based model using GPS analysis for infection spread and inhibition mechanism of SARS-CoV-2 in Tokyo
Authors:
Taishu Murakami,
Shunsuke Sakuragi,
Hiroshi Deguchi,
Masaru Nakata
Abstract:
Analyzing the SARS-CoV-2 pandemic outbreak based on actual data while reflecting the characteristics of the real city provides beneficial information for taking reasonable infection control measures in the future. We demonstrate agent-based modeling for Tokyo based on GPS information and official national statistics and perform a spatiotemporal analysis of the infection situation in Tokyo. As a re…
▽ More
Analyzing the SARS-CoV-2 pandemic outbreak based on actual data while reflecting the characteristics of the real city provides beneficial information for taking reasonable infection control measures in the future. We demonstrate agent-based modeling for Tokyo based on GPS information and official national statistics and perform a spatiotemporal analysis of the infection situation in Tokyo. As a result of the simulation during the first wave of SARS-CoV-2 in Tokyo using real GPS data, the infection occurred in the service industry, such as restaurants, in the city center, and then the infected people brought back the virus to the residential area; the infection spread in each area in Tokyo. This phenomenon clarifies that the spread of infection can be curbed by suppressing going out or strengthening infection prevention measures in service facilities. It was shown that pandemic measures in Tokyo could be achieved not only by strong control, such as the lockdown of cities, but also by thorough infection prevention measures in service facilities, which explains the curb phenomena in real Tokyo.
△ Less
Submitted 26 May, 2022;
originally announced June 2022.
-
Regular generalized solutions to semilinear wave equations
Authors:
Hideo Deguchi,
Michael Oberguggenberger
Abstract:
The paper is devoted to proving an existence and uniqueness result for generalized solutions to semilinear wave equations with a small nonlinearity in space dimensions 1, 2, 3. The setting is the one of Colombeau algebras of generalized functions. It is shown that for a nonlinearity of arbitrary growth and sign, but multiplied with a small parameter, the initial value problem for the semilinear wa…
▽ More
The paper is devoted to proving an existence and uniqueness result for generalized solutions to semilinear wave equations with a small nonlinearity in space dimensions 1, 2, 3. The setting is the one of Colombeau algebras of generalized functions. It is shown that for a nonlinearity of arbitrary growth and sign, but multiplied with a small parameter, the initial value problem for the semilinear wave equation has a unique solution in the Colombeau algebra of generalized functions of bounded type. The proof relies on a fixed point theorem in the ultra-metric topology on the algebras involved. In classical terms, the result says that the semilinear wave equations under consideration have global classical solutions up to a rapidly vanishing error.
△ Less
Submitted 11 September, 2019;
originally announced September 2019.
-
Propagation of singularities for generalized solutions to nonlinear wave equations
Authors:
Hideo Deguchi,
Michael Oberguggenberger
Abstract:
The paper is devoted to regularity theory of generalized solutions to semilinear wave equations with a small nonlinearity. The setting is the one of Colombeau algebras of generalized functions. It is shown that in one space dimension, an initial singularity at the origin propagates along the characteristic lines emanating from the origin, as in the linear case. The proof relies on a fixed point th…
▽ More
The paper is devoted to regularity theory of generalized solutions to semilinear wave equations with a small nonlinearity. The setting is the one of Colombeau algebras of generalized functions. It is shown that in one space dimension, an initial singularity at the origin propagates along the characteristic lines emanating from the origin, as in the linear case. The proof relies on a fixed point theorem in the ultra-metric topology on the algebras involved. The paper takes up the initiating research of the 1970s on anomalous singularities in classical solutions to semilinear hyperbolic equations and transplants the methods into the Colombeau setting.
△ Less
Submitted 16 July, 2019;
originally announced July 2019.
-
Propagation of singularities for generalized solutions to wave equations with discontinuous coefficients
Authors:
Hideo Deguchi,
Michael Oberguggenberger
Abstract:
This article addresses linear hyperbolic partial differential equations with non-smooth coefficients and distributional data. Solutions are studied in the framework of Colombeau algebras of generalized functions. Its aim is to prove upper and lower bounds for the singular support of generalized solutions for wave equations with discontinuous coefficients. New existence results with weaker assumpti…
▽ More
This article addresses linear hyperbolic partial differential equations with non-smooth coefficients and distributional data. Solutions are studied in the framework of Colombeau algebras of generalized functions. Its aim is to prove upper and lower bounds for the singular support of generalized solutions for wave equations with discontinuous coefficients. New existence results with weaker assumptions on the representing families are required and proven. The program is carried through for various types of one- and multidimensional wave equations and hyperbolic systems.
△ Less
Submitted 30 July, 2015;
originally announced July 2015.
-
The wave equation with a discontinuous coefficient depending on time only: generalized solutions and propagation of singularities
Authors:
Hideo Deguchi,
Guenther Hoermann,
Michael Oberguggenberger
Abstract:
This paper is devoted to the investigation of propagation of singularities in hyperbolic equations with non-smooth oefficients, using the Colombeau theory of generalized functions. As a model problem, we study the Cauchy problem for the one-dimensional wave equation with a discontinuous coefficient depending on time. After demonstrating the existence and uniqueness of generalized solutions in the…
▽ More
This paper is devoted to the investigation of propagation of singularities in hyperbolic equations with non-smooth oefficients, using the Colombeau theory of generalized functions. As a model problem, we study the Cauchy problem for the one-dimensional wave equation with a discontinuous coefficient depending on time. After demonstrating the existence and uniqueness of generalized solutions in the sense of Colombeau to the problem, we investigate the phenomenon of propagation of singularities, arising from delta function initial data, for the case of a piecewise constant coefficient. We also provide an analysis of the interplay between singularity strength and propagation effects. Finally, we show that in case the initial data are distributions, the Colombeau solution to the model problem is associated with the piecewise distributional solution of the corresponding transmission problem.
△ Less
Submitted 6 February, 2012;
originally announced February 2012.
