-
Mixing Natural and Synthetic Images for Robust Self-Supervised Representations
Authors:
Reza Akbarian Bafghi,
Nidhin Harilal,
Claire Monteleoni,
Maziar Raissi
Abstract:
This paper introduces DiffMix, a new self-supervised learning (SSL) pre-training framework that combines real and synthetic images. Unlike traditional SSL methods that predominantly use real images, DiffMix uses a variant of Stable Diffusion to replace an augmented instance of a real image, facilitating the learning of cross real-synthetic image representations. The key insight is that while SSL m…
▽ More
This paper introduces DiffMix, a new self-supervised learning (SSL) pre-training framework that combines real and synthetic images. Unlike traditional SSL methods that predominantly use real images, DiffMix uses a variant of Stable Diffusion to replace an augmented instance of a real image, facilitating the learning of cross real-synthetic image representations. The key insight is that while SSL methods trained solely on synthetic images underperform compared to those trained on real images, a blended training approach using both real and synthetic images leads to more robust and adaptable representations. Experiments demonstrate that DiffMix enhances the SSL methods SimCLR, BarlowTwins, and DINO, across various robustness datasets and domain transfer tasks. DiffMix boosts SimCLR's accuracy on ImageNet-1K by 4.56\%. These results challenge the notion that high-quality real images are crucial for SSL pre-training by showing that lower quality synthetic images can also produce strong representations. DiffMix also reduces the need for image augmentations in SSL, offering new optimization strategies.
△ Less
Submitted 18 June, 2024;
originally announced June 2024.
-
Deep LPPLS: Forecasting of temporal critical points in natural, engineering and financial systems
Authors:
Joshua Nielsen,
Didier Sornette,
Maziar Raissi
Abstract:
The Log-Periodic Power Law Singularity (LPPLS) model offers a general framework for capturing dynamics and predicting transition points in diverse natural and social systems. In this work, we present two calibration techniques for the LPPLS model using deep learning. First, we introduce the Mono-LPPLS-NN (M-LNN) model; for any given empirical time series, a unique M-LNN model is trained and shown…
▽ More
The Log-Periodic Power Law Singularity (LPPLS) model offers a general framework for capturing dynamics and predicting transition points in diverse natural and social systems. In this work, we present two calibration techniques for the LPPLS model using deep learning. First, we introduce the Mono-LPPLS-NN (M-LNN) model; for any given empirical time series, a unique M-LNN model is trained and shown to outperform state-of-the-art techniques in estimating the nonlinear parameters $(t_c, m, ω)$ of the LPPLS model as evidenced by the comprehensive distribution of parameter errors. Second, we extend the M-LNN model to a more general model architecture, the Poly-LPPLS-NN (P-LNN), which is able to quickly estimate the nonlinear parameters of the LPPLS model for any given time-series of a fixed length, including previously unseen time-series during training. The Poly class of models train on many synthetic LPPLS time-series augmented with various noise structures in a supervised manner. Given enough training examples, the P-LNN models also outperform state-of-the-art techniques for estimating the parameters of the LPPLS model as evidenced by the comprehensive distribution of parameter errors. Additionally, this class of models is shown to substantially reduce the time to obtain parameter estimates. Finally, we present applications to the diagnostic and prediction of two financial bubble peaks (followed by their crash) and of a famous rockslide. These contributions provide a bridge between deep learning and the study of the prediction of transition times in complex time series.
△ Less
Submitted 21 May, 2024;
originally announced May 2024.
-
Parameter Efficient Fine-tuning of Self-supervised ViTs without Catastrophic Forgetting
Authors:
Reza Akbarian Bafghi,
Nidhin Harilal,
Claire Monteleoni,
Maziar Raissi
Abstract:
Artificial neural networks often suffer from catastrophic forgetting, where learning new concepts leads to a complete loss of previously acquired knowledge. We observe that this issue is particularly magnified in vision transformers (ViTs), where post-pre-training and fine-tuning on new tasks can significantly degrade the model's original general abilities. For instance, a DINO ViT-Base/16 pre-tra…
▽ More
Artificial neural networks often suffer from catastrophic forgetting, where learning new concepts leads to a complete loss of previously acquired knowledge. We observe that this issue is particularly magnified in vision transformers (ViTs), where post-pre-training and fine-tuning on new tasks can significantly degrade the model's original general abilities. For instance, a DINO ViT-Base/16 pre-trained on ImageNet-1k loses over 70% accuracy on ImageNet-1k after just 10 iterations of fine-tuning on CIFAR-100. Overcoming this stability-plasticity dilemma is crucial for enabling ViTs to continuously learn and adapt to new domains while preserving their initial knowledge. In this work, we study two new parameter-efficient fine-tuning strategies: (1)~Block Expansion, and (2) Low-rank adaptation (LoRA). Our experiments reveal that using either Block Expansion or LoRA on self-supervised pre-trained ViTs surpass fully fine-tuned ViTs in new domains while offering significantly greater parameter efficiency. Notably, we find that Block Expansion experiences only a minimal performance drop in the pre-training domain, thereby effectively mitigating catastrophic forgetting in pre-trained ViTs.
△ Less
Submitted 4 July, 2024; v1 submitted 26 April, 2024;
originally announced April 2024.
-
PINNs-TF2: Fast and User-Friendly Physics-Informed Neural Networks in TensorFlow V2
Authors:
Reza Akbarian Bafghi,
Maziar Raissi
Abstract:
Physics-informed neural networks (PINNs) have gained prominence for their capability to tackle supervised learning tasks that conform to physical laws, notably nonlinear partial differential equations (PDEs). This paper presents "PINNs-TF2", a Python package built on the TensorFlow V2 framework. It not only accelerates PINNs implementation but also simplifies user interactions by abstracting compl…
▽ More
Physics-informed neural networks (PINNs) have gained prominence for their capability to tackle supervised learning tasks that conform to physical laws, notably nonlinear partial differential equations (PDEs). This paper presents "PINNs-TF2", a Python package built on the TensorFlow V2 framework. It not only accelerates PINNs implementation but also simplifies user interactions by abstracting complex PDE challenges. We underscore the pivotal role of compilers in PINNs, highlighting their ability to boost performance by up to 119x. Across eight diverse examples, our package, integrated with XLA compilers, demonstrated its flexibility and achieved an average speed-up of 18.12 times over TensorFlow V1. Moreover, a real-world case study is implemented to underscore the compilers' potential to handle many trainable parameters and large batch sizes. For community engagement and future enhancements, our package's source code is openly available at: https://github.com/rezaakb/pinns-tf2.
△ Less
Submitted 6 November, 2023;
originally announced November 2023.
-
A Survey on Physics Informed Reinforcement Learning: Review and Open Problems
Authors:
Chayan Banerjee,
Kien Nguyen,
Clinton Fookes,
Maziar Raissi
Abstract:
The inclusion of physical information in machine learning frameworks has revolutionized many application areas. This involves enhancing the learning process by incorporating physical constraints and adhering to physical laws. In this work we explore their utility for reinforcement learning applications. We present a thorough review of the literature on incorporating physics information, as known a…
▽ More
The inclusion of physical information in machine learning frameworks has revolutionized many application areas. This involves enhancing the learning process by incorporating physical constraints and adhering to physical laws. In this work we explore their utility for reinforcement learning applications. We present a thorough review of the literature on incorporating physics information, as known as physics priors, in reinforcement learning approaches, commonly referred to as physics-informed reinforcement learning (PIRL). We introduce a novel taxonomy with the reinforcement learning pipeline as the backbone to classify existing works, compare and contrast them, and derive crucial insights. Existing works are analyzed with regard to the representation/ form of the governing physics modeled for integration, their specific contribution to the typical reinforcement learning architecture, and their connection to the underlying reinforcement learning pipeline stages. We also identify core learning architectures and physics incorporation biases (i.e., observational, inductive and learning) of existing PIRL approaches and use them to further categorize the works for better understanding and adaptation. By providing a comprehensive perspective on the implementation of the physics-informed capability, the taxonomy presents a cohesive approach to PIRL. It identifies the areas where this approach has been applied, as well as the gaps and opportunities that exist. Additionally, the taxonomy sheds light on unresolved issues and challenges, which can guide future research. This nascent field holds great potential for enhancing reinforcement learning algorithms by increasing their physical plausibility, precision, data efficiency, and applicability in real-world scenarios.
△ Less
Submitted 4 September, 2023;
originally announced September 2023.
-
Physics-Informed Machine Learning of Argon Gas-Driven Melt Pool Dynamics
Authors:
R. Sharma,
W. Grace Guo,
M. Raissi,
Y. B. Guo
Abstract:
Melt pool dynamics in metal additive manufacturing (AM) is critical to process stability, microstructure formation, and final properties of the printed materials. Physics-based simulation including computational fluid dynamics (CFD) is the dominant approach to predict melt pool dynamics. However, the physics-based simulation approaches suffer from the inherent issue of very high computational cost…
▽ More
Melt pool dynamics in metal additive manufacturing (AM) is critical to process stability, microstructure formation, and final properties of the printed materials. Physics-based simulation including computational fluid dynamics (CFD) is the dominant approach to predict melt pool dynamics. However, the physics-based simulation approaches suffer from the inherent issue of very high computational cost. This paper provides a physics-informed machine learning (PIML) method by integrating neural networks with the governing physical laws to predict the melt pool dynamics such as temperature, velocity, and pressure without using any training data on velocity. This approach avoids solving the highly non-linear Navier-Stokes equation numerically, which significantly reduces the computational cost. The difficult-to-determine model constants of the governing equations of the melt pool can also be inferred through data-driven discovery. In addition, the physics-informed neural network (PINN) architecture has been optimized for efficient model training. The data-efficient PINN model is attributed to the soft penalty by incorporating governing partial differential equations (PDEs), initial conditions, and boundary conditions in the PINN model.
△ Less
Submitted 23 July, 2023;
originally announced July 2023.
-
Real Estate Property Valuation using Self-Supervised Vision Transformers
Authors:
Mahdieh Yazdani,
Maziar Raissi
Abstract:
The use of Artificial Intelligence (AI) in the real estate market has been growing in recent years. In this paper, we propose a new method for property valuation that utilizes self-supervised vision transformers, a recent breakthrough in computer vision and deep learning. Our proposed algorithm uses a combination of machine learning, computer vision and hedonic pricing models trained on real estat…
▽ More
The use of Artificial Intelligence (AI) in the real estate market has been growing in recent years. In this paper, we propose a new method for property valuation that utilizes self-supervised vision transformers, a recent breakthrough in computer vision and deep learning. Our proposed algorithm uses a combination of machine learning, computer vision and hedonic pricing models trained on real estate data to estimate the value of a given property. We collected and pre-processed a data set of real estate properties in the city of Boulder, Colorado and used it to train, validate and test our algorithm. Our data set consisted of qualitative images (including house interiors, exteriors, and street views) as well as quantitative features such as the number of bedrooms, bathrooms, square footage, lot square footage, property age, crime rates, and proximity to amenities. We evaluated the performance of our model using metrics such as Root Mean Squared Error (RMSE). Our findings indicate that these techniques are able to accurately predict the value of properties, with a low RMSE. The proposed algorithm outperforms traditional appraisal methods that do not leverage property images and has the potential to be used in real-world applications.
△ Less
Submitted 31 January, 2023;
originally announced February 2023.
-
Temporal Consistency Loss for Physics-Informed Neural Networks
Authors:
Sukirt Thakur,
Maziar Raissi,
Harsa Mitra,
Arezoo Ardekani
Abstract:
Physics-informed neural networks (PINNs) have been widely used to solve partial differential equations in a forward and inverse manner using deep neural networks. However, training these networks can be challenging for multiscale problems. While statistical methods can be employed to scale the regression loss on data, it is generally challenging to scale the loss terms for equations. This paper pr…
▽ More
Physics-informed neural networks (PINNs) have been widely used to solve partial differential equations in a forward and inverse manner using deep neural networks. However, training these networks can be challenging for multiscale problems. While statistical methods can be employed to scale the regression loss on data, it is generally challenging to scale the loss terms for equations. This paper proposes a method for scaling the mean squared loss terms in the objective function used to train PINNs. Instead of using automatic differentiation to calculate the temporal derivative, we use backward Euler discretization. This provides us with a scaling term for the equations. In this work, we consider the two and three-dimensional Navier-Stokes equations and determine the kinematic viscosity using the spatio-temporal data on the velocity and pressure fields. We first consider numerical datasets to test our method. We test the sensitivity of our method to the time step size, the number of timesteps, noise in the data, and spatial resolution. Finally, we use the velocity field obtained using Particle Image Velocimetry (PIV) experiments to generate a reference pressure field. We then test our framework using the velocity and reference pressure field.
△ Less
Submitted 30 January, 2023;
originally announced January 2023.
-
Open Problems in Applied Deep Learning
Authors:
Maziar Raissi
Abstract:
This work formulates the machine learning mechanism as a bi-level optimization problem. The inner level optimization loop entails minimizing a properly chosen loss function evaluated on the training data. This is nothing but the well-studied training process in pursuit of optimal model parameters. The outer level optimization loop is less well-studied and involves maximizing a properly chosen perf…
▽ More
This work formulates the machine learning mechanism as a bi-level optimization problem. The inner level optimization loop entails minimizing a properly chosen loss function evaluated on the training data. This is nothing but the well-studied training process in pursuit of optimal model parameters. The outer level optimization loop is less well-studied and involves maximizing a properly chosen performance metric evaluated on the validation data. This is what we call the "iteration process", pursuing optimal model hyper-parameters. Among many other degrees of freedom, this process entails model engineering (e.g., neural network architecture design) and management, experiment tracking, dataset versioning and augmentation. The iteration process could be automated via Automatic Machine Learning (AutoML) or left to the intuitions of machine learning students, engineers, and researchers. Regardless of the route we take, there is a need to reduce the computational cost of the iteration step and as a direct consequence reduce the carbon footprint of developing artificial intelligence algorithms. Despite the clean and unified mathematical formulation of the iteration step as a bi-level optimization problem, its solutions are case specific and complex. This work will consider such cases while increasing the level of complexity from supervised learning to semi-supervised, self-supervised, unsupervised, few-shot, federated, reinforcement, and physics-informed learning. As a consequence of this exercise, this proposal surfaces a plethora of open problems in the field, many of which can be addressed in parallel.
△ Less
Submitted 26 January, 2023;
originally announced January 2023.
-
Physics-Guided, Physics-Informed, and Physics-Encoded Neural Networks in Scientific Computing
Authors:
Salah A Faroughi,
Nikhil Pawar,
Celio Fernandes,
Maziar Raissi,
Subasish Das,
Nima K. Kalantari,
Seyed Kourosh Mahjour
Abstract:
Recent breakthroughs in computing power have made it feasible to use machine learning and deep learning to advance scientific computing in many fields, including fluid mechanics, solid mechanics, materials science, etc. Neural networks, in particular, play a central role in this hybridization. Due to their intrinsic architecture, conventional neural networks cannot be successfully trained and scop…
▽ More
Recent breakthroughs in computing power have made it feasible to use machine learning and deep learning to advance scientific computing in many fields, including fluid mechanics, solid mechanics, materials science, etc. Neural networks, in particular, play a central role in this hybridization. Due to their intrinsic architecture, conventional neural networks cannot be successfully trained and scoped when data is sparse, which is the case in many scientific and engineering domains. Nonetheless, neural networks provide a solid foundation to respect physics-driven or knowledge-based constraints during training. Generally speaking, there are three distinct neural network frameworks to enforce the underlying physics: (i) physics-guided neural networks (PgNNs), (ii) physics-informed neural networks (PiNNs), and (iii) physics-encoded neural networks (PeNNs). These methods provide distinct advantages for accelerating the numerical modeling of complex multiscale multi-physics phenomena. In addition, the recent developments in neural operators (NOs) add another dimension to these new simulation paradigms, especially when the real-time prediction of complex multi-physics systems is required. All these models also come with their own unique drawbacks and limitations that call for further fundamental research. This study aims to present a review of the four neural network frameworks (i.e., PgNNs, PiNNs, PeNNs, and NOs) used in scientific computing research. The state-of-the-art architectures and their applications are reviewed, limitations are discussed, and future research opportunities in terms of improving algorithms, considering causalities, expanding applications, and coupling scientific and deep learning solvers are presented. This critical review provides researchers and engineers with a solid starting point to comprehend how to integrate different layers of physics into neural networks.
△ Less
Submitted 4 February, 2023; v1 submitted 14 November, 2022;
originally announced November 2022.
-
ViscoelasticNet: A physics informed neural network framework for stress discovery and model selection
Authors:
Sukirt Thakur,
Maziar Raissi,
Arezoo M. Ardekani
Abstract:
Viscoelastic fluids are a class of fluids that exhibit both viscous and elastic nature. Modelling such fluids requires constitutive equations for the stress, and choosing the most appropriate constitutive relationship can be difficult. We present viscoelasticNet, a physics-informed deep learning framework that uses the velocity flow field to select the constitutive model and learn the stress field…
▽ More
Viscoelastic fluids are a class of fluids that exhibit both viscous and elastic nature. Modelling such fluids requires constitutive equations for the stress, and choosing the most appropriate constitutive relationship can be difficult. We present viscoelasticNet, a physics-informed deep learning framework that uses the velocity flow field to select the constitutive model and learn the stress field. Our framework requires data only for the velocity field, initial \& boundary conditions for the stress tensor, and the boundary condition for the pressure field. Using this information, we learn the model parameters, the pressure field, and the stress tensor. {This work considers} three commonly used non-linear viscoelastic models: Oldroyd-B, Giesekus, and linear Phan-Thien-Tanner (PTT). We demonstrate that our framework works well with noisy and sparse data. Our framework can be combined with velocity fields acquired from experimental techniques like particle image velocimetry to get the pressure \& stress fields and model parameters for the constitutive equation. Once the model has been discovered using viscoelasticNet, the fluid can be simulated and modeled for further applications.
△ Less
Submitted 20 June, 2024; v1 submitted 14 September, 2022;
originally announced September 2022.
-
Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's next
Authors:
Salvatore Cuomo,
Vincenzo Schiano di Cola,
Fabio Giampaolo,
Gianluigi Rozza,
Maziar Raissi,
Francesco Piccialli
Abstract:
Physics-Informed Neural Networks (PINN) are neural networks (NNs) that encode model equations, like Partial Differential Equations (PDE), as a component of the neural network itself. PINNs are nowadays used to solve PDEs, fractional equations, integral-differential equations, and stochastic PDEs. This novel methodology has arisen as a multi-task learning framework in which a NN must fit observed d…
▽ More
Physics-Informed Neural Networks (PINN) are neural networks (NNs) that encode model equations, like Partial Differential Equations (PDE), as a component of the neural network itself. PINNs are nowadays used to solve PDEs, fractional equations, integral-differential equations, and stochastic PDEs. This novel methodology has arisen as a multi-task learning framework in which a NN must fit observed data while reducing a PDE residual. This article provides a comprehensive review of the literature on PINNs: while the primary goal of the study was to characterize these networks and their related advantages and disadvantages. The review also attempts to incorporate publications on a broader range of collocation-based physics informed neural networks, which stars form the vanilla PINN, as well as many other variants, such as physics-constrained neural networks (PCNN), variational hp-VPINN, and conservative PINN (CPINN). The study indicates that most research has focused on customizing the PINN through different activation functions, gradient optimization techniques, neural network structures, and loss function structures. Despite the wide range of applications for which PINNs have been used, by demonstrating their ability to be more feasible in some contexts than classical numerical techniques like Finite Element Method (FEM), advancements are still possible, most notably theoretical issues that remain unresolved.
△ Less
Submitted 7 June, 2022; v1 submitted 14 January, 2022;
originally announced January 2022.
-
Data-driven approaches for predicting spread of infectious diseases through DINNs: Disease Informed Neural Networks
Authors:
Sagi Shaier,
Maziar Raissi,
Padmanabhan Seshaiyer
Abstract:
In this work, we present an approach called Disease Informed Neural Networks (DINNs) that can be employed to effectively predict the spread of infectious diseases. This approach builds on a successful physics informed neural network approaches that have been applied to a variety of applications that can be modeled by linear and non-linear ordinary and partial differential equations. Specifically,…
▽ More
In this work, we present an approach called Disease Informed Neural Networks (DINNs) that can be employed to effectively predict the spread of infectious diseases. This approach builds on a successful physics informed neural network approaches that have been applied to a variety of applications that can be modeled by linear and non-linear ordinary and partial differential equations. Specifically, we build on the application of PINNs to SIR compartmental models and expand it a scaffolded family of mathematical models describing various infectious diseases. We show how the neural networks are capable of learning how diseases spread, forecasting their progression, and finding their unique parameters (e.g. death rate). To demonstrate the robustness and efficacy of DINNs, we apply the approach to eleven highly infectious diseases that have been modeled in increasing levels of complexity. Our computational experiments suggest that DINNs is a reliable candidate for effectively learn about the dynamics of spread and forecast its progression into the future from available real-world data.
△ Less
Submitted 24 August, 2022; v1 submitted 11 October, 2021;
originally announced October 2021.
-
A deep learning framework for solution and discovery in solid mechanics
Authors:
Ehsan Haghighat,
Maziar Raissi,
Adrian Moure,
Hector Gomez,
Ruben Juanes
Abstract:
We present the application of a class of deep learning, known as Physics Informed Neural Networks (PINN), to learning and discovery in solid mechanics. We explain how to incorporate the momentum balance and constitutive relations into PINN, and explore in detail the application to linear elasticity, and illustrate its extension to nonlinear problems through an example that showcases von~Mises elas…
▽ More
We present the application of a class of deep learning, known as Physics Informed Neural Networks (PINN), to learning and discovery in solid mechanics. We explain how to incorporate the momentum balance and constitutive relations into PINN, and explore in detail the application to linear elasticity, and illustrate its extension to nonlinear problems through an example that showcases von~Mises elastoplasticity. While common PINN algorithms are based on training one deep neural network (DNN), we propose a multi-network model that results in more accurate representation of the field variables. To validate the model, we test the framework on synthetic data generated from analytical and numerical reference solutions. We study convergence of the PINN model, and show that Isogeometric Analysis (IGA) results in superior accuracy and convergence characteristics compared with classic low-order Finite Element Method (FEM). We also show the applicability of the framework for transfer learning, and find vastly accelerated convergence during network re-training. Finally, we find that honoring the physics leads to improved robustness: when trained only on a few parameters, we find that the PINN model can accurately predict the solution for a wide range of parameters new to the network---thus pointing to an important application of this framework to sensitivity analysis and surrogate modeling.
△ Less
Submitted 6 May, 2020; v1 submitted 14 February, 2020;
originally announced March 2020.
-
Deep Learning of Turbulent Scalar Mixing
Authors:
Maziar Raissi,
Hessam Babaee,
Peyman Givi
Abstract:
Based on recent developments in physics-informed deep learning and deep hidden physics models, we put forth a framework for discovering turbulence models from scattered and potentially noisy spatio-temporal measurements of the probability density function (PDF). The models are for the conditional expected diffusion and the conditional expected dissipation of a Fickian scalar described by its trans…
▽ More
Based on recent developments in physics-informed deep learning and deep hidden physics models, we put forth a framework for discovering turbulence models from scattered and potentially noisy spatio-temporal measurements of the probability density function (PDF). The models are for the conditional expected diffusion and the conditional expected dissipation of a Fickian scalar described by its transported single-point PDF equation. The discovered model are appraised against exact solution derived by the amplitude mapping closure (AMC)/ Johnsohn-Edgeworth translation (JET) model of binary scalar mixing in homogeneous turbulence.
△ Less
Submitted 16 November, 2018;
originally announced November 2018.
-
Deep Learning of Vortex Induced Vibrations
Authors:
Maziar Raissi,
Zhicheng Wang,
Michael S. Triantafyllou,
George Em Karniadakis
Abstract:
Vortex induced vibrations of bluff bodies occur when the vortex shedding frequency is close to the natural frequency of the structure. Of interest is the prediction of the lift and drag forces on the structure given some limited and scattered information on the velocity field. This is an inverse problem that is not straightforward to solve using standard computational fluid dynamics (CFD) methods,…
▽ More
Vortex induced vibrations of bluff bodies occur when the vortex shedding frequency is close to the natural frequency of the structure. Of interest is the prediction of the lift and drag forces on the structure given some limited and scattered information on the velocity field. This is an inverse problem that is not straightforward to solve using standard computational fluid dynamics (CFD) methods, especially since no information is provided for the pressure. An even greater challenge is to infer the lift and drag forces given some dye or smoke visualizations of the flow field. Here we employ deep neural networks that are extended to encode the incompressible Navier-Stokes equations coupled with the structure's dynamic motion equation. In the first case, given scattered data in space-time on the velocity field and the structure's motion, we use four coupled deep neural networks to infer very accurately the structural parameters, the entire time-dependent pressure field (with no prior training data), and reconstruct the velocity vector field and the structure's dynamic motion. In the second case, given scattered data in space-time on a concentration field only, we use five coupled deep neural networks to infer very accurately the vector velocity field and all other quantities of interest as before. This new paradigm of inference in fluid mechanics for coupled multi-physics problems enables velocity and pressure quantification from flow snapshots in small subdomains and can be exploited for flow control applications and also for system identification.
△ Less
Submitted 26 August, 2018;
originally announced August 2018.
-
Hidden Fluid Mechanics: A Navier-Stokes Informed Deep Learning Framework for Assimilating Flow Visualization Data
Authors:
Maziar Raissi,
Alireza Yazdani,
George Em Karniadakis
Abstract:
We present hidden fluid mechanics (HFM), a physics informed deep learning framework capable of encoding an important class of physical laws governing fluid motions, namely the Navier-Stokes equations. In particular, we seek to leverage the underlying conservation laws (i.e., for mass, momentum, and energy) to infer hidden quantities of interest such as velocity and pressure fields merely from spat…
▽ More
We present hidden fluid mechanics (HFM), a physics informed deep learning framework capable of encoding an important class of physical laws governing fluid motions, namely the Navier-Stokes equations. In particular, we seek to leverage the underlying conservation laws (i.e., for mass, momentum, and energy) to infer hidden quantities of interest such as velocity and pressure fields merely from spatio-temporal visualizations of a passive scaler (e.g., dye or smoke), transported in arbitrarily complex domains (e.g., in human arteries or brain aneurysms). Our approach towards solving the aforementioned data assimilation problem is unique as we design an algorithm that is agnostic to the geometry or the initial and boundary conditions. This makes HFM highly flexible in choosing the spatio-temporal domain of interest for data acquisition as well as subsequent training and predictions. Consequently, the predictions made by HFM are among those cases where a pure machine learning strategy or a mere scientific computing approach simply cannot reproduce. The proposed algorithm achieves accurate predictions of the pressure and velocity fields in both two and three dimensional flows for several benchmark problems motivated by real-world applications. Our results demonstrate that this relatively simple methodology can be used in physical and biomedical problems to extract valuable quantitative information (e.g., lift and drag forces or wall shear stresses in arteries) for which direct measurements may not be possible.
△ Less
Submitted 13 August, 2018;
originally announced August 2018.
-
Machine Learning of Space-Fractional Differential Equations
Authors:
Mamikon Gulian,
Maziar Raissi,
Paris Perdikaris,
George Karniadakis
Abstract:
Data-driven discovery of "hidden physics" -- i.e., machine learning of differential equation models underlying observed data -- has recently been approached by embedding the discovery problem into a Gaussian Process regression of spatial data, treating and discovering unknown equation parameters as hyperparameters of a modified "physics informed" Gaussian Process kernel. This kernel includes the p…
▽ More
Data-driven discovery of "hidden physics" -- i.e., machine learning of differential equation models underlying observed data -- has recently been approached by embedding the discovery problem into a Gaussian Process regression of spatial data, treating and discovering unknown equation parameters as hyperparameters of a modified "physics informed" Gaussian Process kernel. This kernel includes the parametrized differential operators applied to a prior covariance kernel. We extend this framework to linear space-fractional differential equations. The methodology is compatible with a wide variety of fractional operators in $\mathbb{R}^d$ and stationary covariance kernels, including the Matern class, and can optimize the Matern parameter during training. We provide a user-friendly and feasible way to perform fractional derivatives of kernels, via a unified set of d-dimensional Fourier integral formulas amenable to generalized Gauss-Laguerre quadrature.
The implementation of fractional derivatives has several benefits. First, it allows for discovering fractional-order PDEs for systems characterized by heavy tails or anomalous diffusion, bypassing the analytical difficulty of fractional calculus. Data sets exhibiting such features are of increasing prevalence in physical and financial domains. Second, a single fractional-order archetype allows for a derivative of arbitrary order to be learned, with the order itself being a parameter in the regression. This is advantageous even when used for discovering integer-order equations; the user is not required to assume a "dictionary" of derivatives of various orders, and directly controls the parsimony of the models being discovered. We illustrate on several examples, including fractional-order interpolation of advection-diffusion and modeling relative stock performance in the S&P 500 with alpha-stable motion via a fractional diffusion equation.
△ Less
Submitted 2 August, 2019; v1 submitted 2 August, 2018;
originally announced August 2018.
-
Cooperative Queuing Policies for Effective Human-Multi-Robot Interaction
Authors:
Masoume M. Raeissi,
Alessandro Farinelli
Abstract:
We consider multi-robot applications, where a team of robots can ask for the intervention of a human operator to handle difficult situations. As the number of requests grows, team members will have to wait for the operator attention, hence the operator becomes a bottleneck for the system. Our aim in this context is to make the robots learn cooperative strategies to decrease the time spent waiting…
▽ More
We consider multi-robot applications, where a team of robots can ask for the intervention of a human operator to handle difficult situations. As the number of requests grows, team members will have to wait for the operator attention, hence the operator becomes a bottleneck for the system. Our aim in this context is to make the robots learn cooperative strategies to decrease the time spent waiting for the operator. In particular, we consider a queuing model where robots decide whether or not to join the queue and use multi-robot learning to estimate the best cooperative policy. In more detail, we formalize the problem as Decentralized Markov Decision Process and provide a suitable state representation, so to apply an independent learners approach. We evaluate the proposed method in a robotic water monitoring simulation and empirically show that our approach can significantly improve the team performance, while being computationally tractable.
△ Less
Submitted 19 June, 2018;
originally announced June 2018.
-
Forward-Backward Stochastic Neural Networks: Deep Learning of High-dimensional Partial Differential Equations
Authors:
Maziar Raissi
Abstract:
Classical numerical methods for solving partial differential equations suffer from the curse dimensionality mainly due to their reliance on meticulously generated spatio-temporal grids. Inspired by modern deep learning based techniques for solving forward and inverse problems associated with partial differential equations, we circumvent the tyranny of numerical discretization by devising an algori…
▽ More
Classical numerical methods for solving partial differential equations suffer from the curse dimensionality mainly due to their reliance on meticulously generated spatio-temporal grids. Inspired by modern deep learning based techniques for solving forward and inverse problems associated with partial differential equations, we circumvent the tyranny of numerical discretization by devising an algorithm that is scalable to high-dimensions. In particular, we approximate the unknown solution by a deep neural network which essentially enables us to benefit from the merits of automatic differentiation. To train the aforementioned neural network we leverage the well-known connection between high-dimensional partial differential equations and forward-backward stochastic differential equations. In fact, independent realizations of a standard Brownian motion will act as training data. We test the effectiveness of our approach for a couple of benchmark problems spanning a number of scientific domains including Black-Scholes-Barenblatt and Hamilton-Jacobi-Bellman equations, both in 100-dimensions.
△ Less
Submitted 19 April, 2018;
originally announced April 2018.
-
Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations
Authors:
Maziar Raissi
Abstract:
A long-standing problem at the interface of artificial intelligence and applied mathematics is to devise an algorithm capable of achieving human level or even superhuman proficiency in transforming observed data into predictive mathematical models of the physical world. In the current era of abundance of data and advanced machine learning capabilities, the natural question arises: How can we autom…
▽ More
A long-standing problem at the interface of artificial intelligence and applied mathematics is to devise an algorithm capable of achieving human level or even superhuman proficiency in transforming observed data into predictive mathematical models of the physical world. In the current era of abundance of data and advanced machine learning capabilities, the natural question arises: How can we automatically uncover the underlying laws of physics from high-dimensional data generated from experiments? In this work, we put forth a deep learning approach for discovering nonlinear partial differential equations from scattered and potentially noisy observations in space and time. Specifically, we approximate the unknown solution as well as the nonlinear dynamics by two deep neural networks. The first network acts as a prior on the unknown solution and essentially enables us to avoid numerical differentiations which are inherently ill-conditioned and unstable. The second network represents the nonlinear dynamics and helps us distill the mechanisms that govern the evolution of a given spatiotemporal data-set. We test the effectiveness of our approach for several benchmark problems spanning a number of scientific domains and demonstrate how the proposed framework can help us accurately learn the underlying dynamics and forecast future states of the system. In particular, we study the Burgers', Korteweg-de Vries (KdV), Kuramoto-Sivashinsky, nonlinear Schrödinger, and Navier-Stokes equations.
△ Less
Submitted 20 January, 2018;
originally announced January 2018.
-
Multistep Neural Networks for Data-driven Discovery of Nonlinear Dynamical Systems
Authors:
Maziar Raissi,
Paris Perdikaris,
George Em Karniadakis
Abstract:
The process of transforming observed data into predictive mathematical models of the physical world has always been paramount in science and engineering. Although data is currently being collected at an ever-increasing pace, devising meaningful models out of such observations in an automated fashion still remains an open problem. In this work, we put forth a machine learning approach for identifyi…
▽ More
The process of transforming observed data into predictive mathematical models of the physical world has always been paramount in science and engineering. Although data is currently being collected at an ever-increasing pace, devising meaningful models out of such observations in an automated fashion still remains an open problem. In this work, we put forth a machine learning approach for identifying nonlinear dynamical systems from data. Specifically, we blend classical tools from numerical analysis, namely the multi-step time-stepping schemes, with powerful nonlinear function approximators, namely deep neural networks, to distill the mechanisms that govern the evolution of a given data-set. We test the effectiveness of our approach for several benchmark problems involving the identification of complex, nonlinear and chaotic dynamics, and we demonstrate how this allows us to accurately learn the dynamics, forecast future states, and identify basins of attraction. In particular, we study the Lorenz system, the fluid flow behind a cylinder, the Hopf bifurcation, and the Glycoltic oscillator model as an example of complicated nonlinear dynamics typical of biological systems.
△ Less
Submitted 3 January, 2018;
originally announced January 2018.
-
Physics Informed Deep Learning (Part II): Data-driven Discovery of Nonlinear Partial Differential Equations
Authors:
Maziar Raissi,
Paris Perdikaris,
George Em Karniadakis
Abstract:
We introduce physics informed neural networks -- neural networks that are trained to solve supervised learning tasks while respecting any given law of physics described by general nonlinear partial differential equations. In this second part of our two-part treatise, we focus on the problem of data-driven discovery of partial differential equations. Depending on whether the available data is scatt…
▽ More
We introduce physics informed neural networks -- neural networks that are trained to solve supervised learning tasks while respecting any given law of physics described by general nonlinear partial differential equations. In this second part of our two-part treatise, we focus on the problem of data-driven discovery of partial differential equations. Depending on whether the available data is scattered in space-time or arranged in fixed temporal snapshots, we introduce two main classes of algorithms, namely continuous time and discrete time models. The effectiveness of our approach is demonstrated using a wide range of benchmark problems in mathematical physics, including conservation laws, incompressible fluid flow, and the propagation of nonlinear shallow-water waves.
△ Less
Submitted 28 November, 2017;
originally announced November 2017.
-
Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations
Authors:
Maziar Raissi,
Paris Perdikaris,
George Em Karniadakis
Abstract:
We introduce physics informed neural networks -- neural networks that are trained to solve supervised learning tasks while respecting any given law of physics described by general nonlinear partial differential equations. In this two part treatise, we present our developments in the context of solving two main classes of problems: data-driven solution and data-driven discovery of partial different…
▽ More
We introduce physics informed neural networks -- neural networks that are trained to solve supervised learning tasks while respecting any given law of physics described by general nonlinear partial differential equations. In this two part treatise, we present our developments in the context of solving two main classes of problems: data-driven solution and data-driven discovery of partial differential equations. Depending on the nature and arrangement of the available data, we devise two distinct classes of algorithms, namely continuous time and discrete time models. The resulting neural networks form a new class of data-efficient universal function approximators that naturally encode any underlying physical laws as prior information. In this first part, we demonstrate how these networks can be used to infer solutions to partial differential equations, and obtain physics-informed surrogate models that are fully differentiable with respect to all input coordinates and free parameters.
△ Less
Submitted 28 November, 2017;
originally announced November 2017.
-
Hidden Physics Models: Machine Learning of Nonlinear Partial Differential Equations
Authors:
Maziar Raissi,
George Em Karniadakis
Abstract:
While there is currently a lot of enthusiasm about "big data", useful data is usually "small" and expensive to acquire. In this paper, we present a new paradigm of learning partial differential equations from {\em small} data. In particular, we introduce \emph{hidden physics models}, which are essentially data-efficient learning machines capable of leveraging the underlying laws of physics, expres…
▽ More
While there is currently a lot of enthusiasm about "big data", useful data is usually "small" and expensive to acquire. In this paper, we present a new paradigm of learning partial differential equations from {\em small} data. In particular, we introduce \emph{hidden physics models}, which are essentially data-efficient learning machines capable of leveraging the underlying laws of physics, expressed by time dependent and nonlinear partial differential equations, to extract patterns from high-dimensional data generated from experiments. The proposed methodology may be applied to the problem of learning, system identification, or data-driven discovery of partial differential equations. Our framework relies on Gaussian processes, a powerful tool for probabilistic inference over functions, that enables us to strike a balance between model complexity and data fitting. The effectiveness of the proposed approach is demonstrated through a variety of canonical problems, spanning a number of scientific domains, including the Navier-Stokes, Schrödinger, Kuramoto-Sivashinsky, and time dependent linear fractional equations. The methodology provides a promising new direction for harnessing the long-standing developments of classical methods in applied mathematics and mathematical physics to design learning machines with the ability to operate in complex domains without requiring large quantities of data.
△ Less
Submitted 21 August, 2017; v1 submitted 1 August, 2017;
originally announced August 2017.
-
Parametric Gaussian Process Regression for Big Data
Authors:
Maziar Raissi
Abstract:
This work introduces the concept of parametric Gaussian processes (PGPs), which is built upon the seemingly self-contradictory idea of making Gaussian processes parametric. Parametric Gaussian processes, by construction, are designed to operate in "big data" regimes where one is interested in quantifying the uncertainty associated with noisy data. The proposed methodology circumvents the well-esta…
▽ More
This work introduces the concept of parametric Gaussian processes (PGPs), which is built upon the seemingly self-contradictory idea of making Gaussian processes parametric. Parametric Gaussian processes, by construction, are designed to operate in "big data" regimes where one is interested in quantifying the uncertainty associated with noisy data. The proposed methodology circumvents the well-established need for stochastic variational inference, a scalable algorithm for approximating posterior distributions. The effectiveness of the proposed approach is demonstrated using an illustrative example with simulated data and a benchmark dataset in the airline industry with approximately 6 million records.
△ Less
Submitted 4 May, 2017; v1 submitted 11 April, 2017;
originally announced April 2017.
-
Numerical Gaussian Processes for Time-dependent and Non-linear Partial Differential Equations
Authors:
Maziar Raissi,
Paris Perdikaris,
George Em Karniadakis
Abstract:
We introduce the concept of numerical Gaussian processes, which we define as Gaussian processes with covariance functions resulting from temporal discretization of time-dependent partial differential equations. Numerical Gaussian processes, by construction, are designed to deal with cases where: (1) all we observe are noisy data on black-box initial conditions, and (2) we are interested in quantif…
▽ More
We introduce the concept of numerical Gaussian processes, which we define as Gaussian processes with covariance functions resulting from temporal discretization of time-dependent partial differential equations. Numerical Gaussian processes, by construction, are designed to deal with cases where: (1) all we observe are noisy data on black-box initial conditions, and (2) we are interested in quantifying the uncertainty associated with such noisy data in our solutions to time-dependent partial differential equations. Our method circumvents the need for spatial discretization of the differential operators by proper placement of Gaussian process priors. This is an attempt to construct structured and data-efficient learning machines, which are explicitly informed by the underlying physics that possibly generated the observed data. The effectiveness of the proposed approach is demonstrated through several benchmark problems involving linear and nonlinear time-dependent operators. In all examples, we are able to recover accurate approximations of the latent solutions, and consistently propagate uncertainty, even in cases involving very long time integration.
△ Less
Submitted 29 March, 2017;
originally announced March 2017.
-
Machine Learning of Linear Differential Equations using Gaussian Processes
Authors:
Maziar Raissi,
George Em. Karniadakis
Abstract:
This work leverages recent advances in probabilistic machine learning to discover conservation laws expressed by parametric linear equations. Such equations involve, but are not limited to, ordinary and partial differential, integro-differential, and fractional order operators. Here, Gaussian process priors are modified according to the particular form of such operators and are employed to infer p…
▽ More
This work leverages recent advances in probabilistic machine learning to discover conservation laws expressed by parametric linear equations. Such equations involve, but are not limited to, ordinary and partial differential, integro-differential, and fractional order operators. Here, Gaussian process priors are modified according to the particular form of such operators and are employed to infer parameters of the linear equations from scarce and possibly noisy observations. Such observations may come from experiments or "black-box" computer simulations.
△ Less
Submitted 10 January, 2017;
originally announced January 2017.
-
Inferring solutions of differential equations using noisy multi-fidelity data
Authors:
Maziar Raissi,
Paris Perdikaris,
George Em. Karniadakis
Abstract:
For more than two centuries, solutions of differential equations have been obtained either analytically or numerically based on typically well-behaved forcing and boundary conditions for well-posed problems. We are changing this paradigm in a fundamental way by establishing an interface between probabilistic machine learning and differential equations. We develop data-driven algorithms for general…
▽ More
For more than two centuries, solutions of differential equations have been obtained either analytically or numerically based on typically well-behaved forcing and boundary conditions for well-posed problems. We are changing this paradigm in a fundamental way by establishing an interface between probabilistic machine learning and differential equations. We develop data-driven algorithms for general linear equations using Gaussian process priors tailored to the corresponding integro-differential operators. The only observables are scarce noisy multi-fidelity data for the forcing and solution that are not required to reside on the domain boundary. The resulting predictive posterior distributions quantify uncertainty and naturally lead to adaptive solution refinement via active learning. This general framework circumvents the tyranny of numerical discretization as well as the consistency and stability issues of time-integration, and is scalable to high-dimensions.
△ Less
Submitted 16 July, 2016;
originally announced July 2016.
-
Deep Multi-fidelity Gaussian Processes
Authors:
Maziar Raissi,
George Karniadakis
Abstract:
We develop a novel multi-fidelity framework that goes far beyond the classical AR(1) Co-kriging scheme of Kennedy and O'Hagan (2000). Our method can handle general discontinuous cross-correlations among systems with different levels of fidelity. A combination of multi-fidelity Gaussian Processes (AR(1) Co-kriging) and deep neural networks enables us to construct a method that is immune to disconti…
▽ More
We develop a novel multi-fidelity framework that goes far beyond the classical AR(1) Co-kriging scheme of Kennedy and O'Hagan (2000). Our method can handle general discontinuous cross-correlations among systems with different levels of fidelity. A combination of multi-fidelity Gaussian Processes (AR(1) Co-kriging) and deep neural networks enables us to construct a method that is immune to discontinuities. We demonstrate the effectiveness of the new technology using standard benchmark problems designed to resemble the outputs of complicated high- and low-fidelity codes.
△ Less
Submitted 25 April, 2016;
originally announced April 2016.
-
Numerical results a quantum waveguide with Mixed boundary conditions
Authors:
M. Raissi
Abstract:
This article is devoted to the numerical study of the existence of the eigenvalues of the Hamiltonian describing a quantum particle living on three dimensional straight strip of width $d$ in the presence of an electric field of constant intensity $F$ in the direction perpendicular to the electron plane. We impose Neumann boundary conditions on a disc window of radius $a$ and Dirichlet boundary con…
▽ More
This article is devoted to the numerical study of the existence of the eigenvalues of the Hamiltonian describing a quantum particle living on three dimensional straight strip of width $d$ in the presence of an electric field of constant intensity $F$ in the direction perpendicular to the electron plane. We impose Neumann boundary conditions on a disc window of radius $a$ and Dirichlet boundary conditions on the remaining part of the boundary of the strip.
△ Less
Submitted 6 September, 2019; v1 submitted 9 January, 2015;
originally announced January 2015.
-
A multi-fidelity stochastic collocation method for parabolic PDEs with random input data
Authors:
Maziar Raissi,
Padmanabhan Seshaiyer
Abstract:
Over the last few years there have been dramatic advances in our understanding of mathematical and computational models of complex systems in the presence of uncertainty. This has led to a growth in the area of uncertainty quantification as well as the need to develop efficient, scalable, stable and convergent computational methods for solving differential equations with random inputs. Stochastic…
▽ More
Over the last few years there have been dramatic advances in our understanding of mathematical and computational models of complex systems in the presence of uncertainty. This has led to a growth in the area of uncertainty quantification as well as the need to develop efficient, scalable, stable and convergent computational methods for solving differential equations with random inputs. Stochastic Galerkin methods based on polynomial chaos expansions have shown superiority to other non-sampling and many sampling techniques. However, for complicated governing equations numerical implementations of stochastic Galerkin methods can become non-trivial. On the other hand, Monte Carlo and other traditional sampling methods, are straightforward to implement. However, they do not offer as fast convergence rates as stochastic Galerkin. Other numerical approaches are the stochastic collocation (SC) methods, which inherit both, the ease of implementation of Monte Carlo and the robustness of stochastic Galerkin to a great deal. In this work we propose a novel enhancement to stochastic collocation methods using deterministic model reduction techniques. Linear parabolic partial differential equations with random forcing terms are analysed. The input data are assumed to be represented by a finite number of random variables. A rigorous convergence analysis, supported by numerical results, shows that the proposed technique is not only reliable and robust but also efficient.
△ Less
Submitted 1 June, 2013;
originally announced June 2013.
-
A multi-fidelity stochastic collocation method using locally improved reduced-order models
Authors:
Maziar Raissi,
Padmanabhan Seshaiyer
Abstract:
Over the last few years there have been dramatic advances in our understanding of mathematical and computational models of complex systems in the presence of uncertainty. This has led to a growth in the area of uncertainty quantification as well as the need to develop efficient, scalable, stable and convergent computational methods for solving differential equations with random inputs. Stochastic…
▽ More
Over the last few years there have been dramatic advances in our understanding of mathematical and computational models of complex systems in the presence of uncertainty. This has led to a growth in the area of uncertainty quantification as well as the need to develop efficient, scalable, stable and convergent computational methods for solving differential equations with random inputs. Stochastic Galerkin methods based on polynomial chaos expansions have shown superiority to other non-sampling and many sampling techniques. However, for complicated governing equations numerical implementations of stochastic Galerkin methods can become non-trivial. On the other hand, Monte Carlo and other traditional sampling methods, are straightforward to implement. However, they do not offer as fast convergence rates as stochastic Galerkin. Other numerical approaches are the stochastic collocation (SC) methods, which inherit both, the ease of implementation of Monte Carlo and the robustness of stochastic Galerkin to a great deal. However, stochastic collocation and its powerful extensions, e.g. sparse grid stochastic collocation, can simply fail to handle more levels of complication. The seemingly innocent Burgers equation driven by Brownian motion is such an example. In this work we propose a novel enhancement to stochastic collocation methods using locally improved deterministic model reduction techniques that can handle this pathological example and hopefully other more complicated equations like Stochastic Navier-Stokes. Local improvements to reduced-order models are achieved using sensitivity analysis of the proper orthogonal decomposition. Our numerical results show that the proposed technique is not only reliable and robust but also very efficient.
△ Less
Submitted 1 June, 2013;
originally announced June 2013.
-
A quantum waveguide with Aharonov Bohm magnetic field
Authors:
H. Najar,
M. Raissi
Abstract:
In a previous study \cite{n} we investigate the bound states of the Hamiltonian describing a quantum particle living on three dimensional straight strip of width $d$. We impose the Neumann boundary condition on a disc window of radius $a$ and Dirichlet boundary conditions on the remained part of the boundary of the strip. We proved that such system exhibits discrete eigenvalues below the essential…
▽ More
In a previous study \cite{n} we investigate the bound states of the Hamiltonian describing a quantum particle living on three dimensional straight strip of width $d$. We impose the Neumann boundary condition on a disc window of radius $a$ and Dirichlet boundary conditions on the remained part of the boundary of the strip. We proved that such system exhibits discrete eigenvalues below the essential spectrum for any $a>0$. In the present work we study the effect of a magnetic filed of Aharonov-Bohm type when the magnetic field is turned on this system. Precisely we prove that in the presence of such magnetic filed there is some critical values of $a_0>0$, for which we have absence of the discrete spectrum for $\displaystyle 0<\frac{a}{d}<a_0$. We give a sufficient condition for the existence of discrete eigenvalues.
△ Less
Submitted 21 February, 2016; v1 submitted 10 November, 2012;
originally announced November 2012.
