Showing 1–50 of 84 results for author: Karniadakis, G E

Multiscale modeling framework of a constrained fluid with complex boundaries using twin neural networks

Authors: Peiyuan Gao, George Em Karniadakis, Panos Stinis

Abstract: The properties of constrained fluids have increasingly gained relevance for applications ranging from materials to biology. In this work, we propose a multiscale model using twin neural networks to investigate the properties of a fluid constrained between solid surfaces with complex shapes. The atomic scale model and the mesoscale model are connected by the coarse-grained potential which is repres… ▽ More The properties of constrained fluids have increasingly gained relevance for applications ranging from materials to biology. In this work, we propose a multiscale model using twin neural networks to investigate the properties of a fluid constrained between solid surfaces with complex shapes. The atomic scale model and the mesoscale model are connected by the coarse-grained potential which is represented by the first neural network. Then we train the second neural network model as a surrogate to predict the velocity profile of the constrained fluid with complex boundary conditions at the mesoscale. The effect of complex boundary conditions on the fluid dynamics properties and the accuracy of the neural network model prediction are systematically investigated. We demonstrate that the neural network-enhanced multiscale framework can connect simulations at atomic scale and mesoscale and reproduce the properties of a constrained fluid at mesoscale. This work provides insight into multiscale model development with the aid of machine learning techniques and the developed model can be used for modern nanotechnology applications such as enhanced oil recovery and porous materials design. △ Less

Submitted 6 August, 2024; originally announced August 2024.

NeuroSEM: A hybrid framework for simulating multiphysics problems by coupling PINNs and spectral elements

Authors: Khemraj Shukla, Zongren Zou, Chi Hin Chan, Additi Pandey, Zhicheng Wang, George Em Karniadakis

Abstract: Multiphysics problems that are characterized by complex interactions among fluid dynamics, heat transfer, structural mechanics, and electromagnetics, are inherently challenging due to their coupled nature. While experimental data on certain state variables may be available, integrating these data with numerical solvers remains a significant challenge. Physics-informed neural networks (PINNs) have… ▽ More Multiphysics problems that are characterized by complex interactions among fluid dynamics, heat transfer, structural mechanics, and electromagnetics, are inherently challenging due to their coupled nature. While experimental data on certain state variables may be available, integrating these data with numerical solvers remains a significant challenge. Physics-informed neural networks (PINNs) have shown promising results in various engineering disciplines, particularly in handling noisy data and solving inverse problems. However, their effectiveness in forecasting nonlinear phenomena in multiphysics regimes is yet to be fully established. This study introduces NeuroSEM, a hybrid framework integrating PINNs with the high-fidelity Spectral Element Method (SEM) solver, Nektar++. NeuroSEM leverages strengths of both PINNs and SEM, providing robust solutions for multiphysics problems. PINNs are trained to assimilate data and model physical phenomena in specific subdomains, which are then integrated into Nektar++. We demonstrate the efficiency and accuracy of NeuroSEM for thermal convection in cavity flow and flow past a cylinder. The framework effectively handles data assimilation by addressing those subdomains and state variables where data are available. We applied NeuroSEM to the Rayleigh-Bénard convection system, including cases with missing thermal boundary conditions. Our results indicate that NeuroSEM accurately models the physical phenomena and assimilates the data within the specified subdomains. The framework's plug-and-play nature facilitates its extension to other multiphysics or multiscale problems. Furthermore, NeuroSEM is optimized for an efficient execution on emerging integrated GPU-CPU architectures. This hybrid approach enhances the accuracy and efficiency of simulations, making it a powerful tool for tackling complex engineering challenges in various scientific domains. △ Less

Submitted 30 July, 2024; originally announced July 2024.

Inferring turbulent velocity and temperature fields and their statistics from Lagrangian velocity measurements using physics-informed Kolmogorov-Arnold Networks

Authors: Juan Diego Toscano, Theo Käufer, Zhibo Wang, Martin Maxey, Christian Cierpka, George Em Karniadakis

Abstract: We propose the Artificial Intelligence Velocimetry-Thermometry (AIVT) method to infer hidden temperature fields from experimental turbulent velocity data. This physics-informed machine learning method enables us to infer continuous temperature fields using only sparse velocity data, hence eliminating the need for direct temperature measurements. Specifically, AIVT is based on physics-informed Kolm… ▽ More We propose the Artificial Intelligence Velocimetry-Thermometry (AIVT) method to infer hidden temperature fields from experimental turbulent velocity data. This physics-informed machine learning method enables us to infer continuous temperature fields using only sparse velocity data, hence eliminating the need for direct temperature measurements. Specifically, AIVT is based on physics-informed Kolmogorov-Arnold Networks (not neural networks) and is trained by optimizing a combined loss function that minimizes the residuals of the velocity data, boundary conditions, and the governing equations. We apply AIVT to a unique set of experimental volumetric and simultaneous temperature and velocity data of Rayleigh-Bénard convection (RBC) that we acquired by combining Particle Image Thermometry and Lagrangian Particle Tracking. This allows us to compare AIVT predictions and measurements directly. We demonstrate that we can reconstruct and infer continuous and instantaneous velocity and temperature fields from sparse experimental data at a fidelity comparable to direct numerical simulations (DNS) of turbulence. This, in turn, enables us to compute important quantities for quantifying turbulence, such as fluctuations, viscous and thermal dissipation, and QR distribution. This paradigm shift in processing experimental data using AIVT to infer turbulent fields at DNS-level fidelity is a promising avenue in breaking the current deadlock of quantitative understanding of turbulence at high Reynolds numbers, where DNS is computationally infeasible. △ Less

Submitted 23 July, 2024; v1 submitted 22 July, 2024; originally announced July 2024.

Comments: turbulence, data assimilation, physics-informed machine learning, experimental methods, Kolmogorov-Arnold networks. 50 pages, 8 figures

A comprehensive and FAIR comparison between MLP and KAN representations for differential equations and operator networks

Authors: Khemraj Shukla, Juan Diego Toscano, Zhicheng Wang, Zongren Zou, George Em Karniadakis

Abstract: Kolmogorov-Arnold Networks (KANs) were recently introduced as an alternative representation model to MLP. Herein, we employ KANs to construct physics-informed machine learning models (PIKANs) and deep operator models (DeepOKANs) for solving differential equations for forward and inverse problems. In particular, we compare them with physics-informed neural networks (PINNs) and deep operator network… ▽ More Kolmogorov-Arnold Networks (KANs) were recently introduced as an alternative representation model to MLP. Herein, we employ KANs to construct physics-informed machine learning models (PIKANs) and deep operator models (DeepOKANs) for solving differential equations for forward and inverse problems. In particular, we compare them with physics-informed neural networks (PINNs) and deep operator networks (DeepONets), which are based on the standard MLP representation. We find that although the original KANs based on the B-splines parameterization lack accuracy and efficiency, modified versions based on low-order orthogonal polynomials have comparable performance to PINNs and DeepONet although they still lack robustness as they may diverge for different random seeds or higher order orthogonal polynomials. We visualize their corresponding loss landscapes and analyze their learning dynamics using information bottleneck theory. Our study follows the FAIR principles so that other researchers can use our benchmarks to further advance this emerging topic. △ Less

Submitted 5 June, 2024; originally announced June 2024.

Large scale scattering using fast solvers based on neural operators

Authors: Zongren Zou, Adar Kahana, Enrui Zhang, Eli Turkel, Rishikesh Ranade, Jay Pathak, George Em Karniadakis

Abstract: We extend a recently proposed machine-learning-based iterative solver, i.e. the hybrid iterative transferable solver (HINTS), to solve the scattering problem described by the Helmholtz equation in an exterior domain with a complex absorbing boundary condition. The HINTS method combines neural operators (NOs) with standard iterative solvers, e.g. Jacobi and Gauss-Seidel (GS), to achieve better perf… ▽ More We extend a recently proposed machine-learning-based iterative solver, i.e. the hybrid iterative transferable solver (HINTS), to solve the scattering problem described by the Helmholtz equation in an exterior domain with a complex absorbing boundary condition. The HINTS method combines neural operators (NOs) with standard iterative solvers, e.g. Jacobi and Gauss-Seidel (GS), to achieve better performance by leveraging the spectral bias of neural networks. In HINTS, some iterations of the conventional iterative method are replaced by inferences of the pre-trained NO. In this work, we employ HINTS to solve the scattering problem for both 2D and 3D problems, where the standard iterative solver fails. We consider square and triangular scatterers of various sizes in 2D, and a cube and a model submarine in 3D. We explore and illustrate the extrapolation capability of HINTS in handling diverse geometries of the scatterer, which is achieved by training the NO on non-scattering scenarios and then deploying it in HINTS to solve scattering problems. The accurate results demonstrate that the NO in HINTS method remains effective without retraining or fine-tuning it whenever a new scatterer is given. Taken together, our results highlight the adaptability and versatility of the extended HINTS methodology in addressing diverse scattering problems. △ Less

Submitted 20 May, 2024; originally announced May 2024.

Two-scale Neural Networks for Partial Differential Equations with Small Parameters

Authors: Qiao Zhuang, Chris Ziyi Yao, Zhongqiang Zhang, George Em Karniadakis

Abstract: We propose a two-scale neural network method for solving partial differential equations (PDEs) with small parameters using physics-informed neural networks (PINNs). We directly incorporate the small parameters into the architecture of neural networks. The proposed method enables solving PDEs with small parameters in a simple fashion, without adding Fourier features or other computationally taxing… ▽ More We propose a two-scale neural network method for solving partial differential equations (PDEs) with small parameters using physics-informed neural networks (PINNs). We directly incorporate the small parameters into the architecture of neural networks. The proposed method enables solving PDEs with small parameters in a simple fashion, without adding Fourier features or other computationally taxing searches of truncation parameters. Various numerical examples demonstrate reasonable accuracy in capturing features of large derivatives in the solutions caused by small parameters. △ Less

Submitted 27 February, 2024; originally announced February 2024.

MSC Class: 65N35; 35B25 ACM Class: I.2.6

RiemannONets: Interpretable Neural Operators for Riemann Problems

Authors: Ahmad Peyvan, Vivek Oommen, Ameya D. Jagtap, George Em Karniadakis

Abstract: Developing the proper representations for simulating high-speed flows with strong shock waves, rarefactions, and contact discontinuities has been a long-standing question in numerical analysis. Herein, we employ neural operators to solve Riemann problems encountered in compressible flows for extreme pressure jumps (up to $10^{10}$ pressure ratio). In particular, we first consider the DeepONet that… ▽ More Developing the proper representations for simulating high-speed flows with strong shock waves, rarefactions, and contact discontinuities has been a long-standing question in numerical analysis. Herein, we employ neural operators to solve Riemann problems encountered in compressible flows for extreme pressure jumps (up to $10^{10}$ pressure ratio). In particular, we first consider the DeepONet that we train in a two-stage process, following the recent work of \cite{lee2023training}, wherein the first stage, a basis is extracted from the trunk net, which is orthonormalized and subsequently is used in the second stage in training the branch net. This simple modification of DeepONet has a profound effect on its accuracy, efficiency, and robustness and leads to very accurate solutions to Riemann problems compared to the vanilla version. It also enables us to interpret the results physically as the hierarchical data-driven produced basis reflects all the flow features that would otherwise be introduced using ad hoc feature expansion layers. We also compare the results with another neural operator based on the U-Net for low, intermediate, and very high-pressure ratios that are very accurate for Riemann problems, especially for large pressure ratios, due to their multiscale nature but computationally more expensive. Overall, our study demonstrates that simple neural network architectures, if properly pre-trained, can achieve very accurate solutions of Riemann problems for real-time forecasting. The source code, along with its corresponding data, can be found at the following URL: https://github.com/apey236/RiemannONet/tree/main △ Less

Submitted 16 April, 2024; v1 submitted 16 January, 2024; originally announced January 2024.

Learning thermoacoustic interactions in combustors using a physics-informed neural network

Authors: Sathesh Mariappan, Kamaljyoti Nath, George Em Karniadakis

Abstract: We introduce a physics-informed neural network (PINN) method to study thermoacoustic interactions leading to combustion instability in combustors. Specifically, we employ a PINN to investigate thermoacoustic interactions in a bluff body anchored flame combustor, representative of ramjet and industrial combustors. Vortex shedding and acoustic oscillations appear in such combustors, and their intera… ▽ More We introduce a physics-informed neural network (PINN) method to study thermoacoustic interactions leading to combustion instability in combustors. Specifically, we employ a PINN to investigate thermoacoustic interactions in a bluff body anchored flame combustor, representative of ramjet and industrial combustors. Vortex shedding and acoustic oscillations appear in such combustors, and their interactions lead to the phenomenon of vortex-acoustic lock-in. Acoustic pressure fluctuations at three locations and the total flame heat release rate serve as the measured data. The coupled parameterized model is based on the acoustic equations and the van der Pol oscillator for vortex shedding. The PINN was applied in the combustor, where the measurements suitable for a future machine learning application were not anticipated at the time of the experiments, as is the case in the vast majority of available data in the literature. We demonstrate a good performance of PINN in generating the acoustic field (pressure and velocity fluctuations) in the entire spatiotemporal domain, along with estimating all the parameters of the model. Therefore, this PINN-based model can potentially serve as an effective tool in improving existing combustors or designing new thermoacoustically stable and structurally efficient combustors. △ Less

Submitted 29 December, 2023; originally announced January 2024.

AI-Lorenz: A physics-data-driven framework for black-box and gray-box identification of chaotic systems with symbolic regression

Authors: Mario De Florio, Ioannis G. Kevrekidis, George Em Karniadakis

Abstract: Discovering mathematical models that characterize the observed behavior of dynamical systems remains a major challenge, especially for systems in a chaotic regime. The challenge is even greater when the physics underlying such systems is not yet understood, and scientific inquiry must solely rely on empirical data. Driven by the need to fill this gap, we develop a framework that learns mathematica… ▽ More Discovering mathematical models that characterize the observed behavior of dynamical systems remains a major challenge, especially for systems in a chaotic regime. The challenge is even greater when the physics underlying such systems is not yet understood, and scientific inquiry must solely rely on empirical data. Driven by the need to fill this gap, we develop a framework that learns mathematical expressions modeling complex dynamical behaviors by identifying differential equations from noisy and sparse observable data. We train a small neural network to learn the dynamics of a system, its rate of change in time, and missing model terms, which are used as input for a symbolic regression algorithm to autonomously distill the explicit mathematical terms. This, in turn, enables us to predict the future evolution of the dynamical behavior. The performance of this framework is validated by recovering the right-hand sides and unknown terms of certain complex, chaotic systems such as the well-known Lorenz system, a six-dimensional hyperchaotic system, and the non-autonomous Sprott chaotic system, and comparing them with their known analytical expressions. △ Less

Submitted 21 December, 2023; originally announced December 2023.

Comments: 28 pages, 15 figures, 9 tables

MSC Class: 34A34; 34A55; 70K55 ACM Class: J.2; G.1.7; I.2.0

Rethinking materials simulations: Blending direct numerical simulations with neural operators

Authors: Vivek Oommen, Khemraj Shukla, Saaketh Desai, Remi Dingreville, George Em Karniadakis

Abstract: Direct numerical simulations (DNS) are accurate but computationally expensive for predicting materials evolution across timescales, due to the complexity of the underlying evolution equations, the nature of multiscale spatio-temporal interactions, and the need to reach long-time integration. We develop a new method that blends numerical solvers with neural operators to accelerate such simulations.… ▽ More Direct numerical simulations (DNS) are accurate but computationally expensive for predicting materials evolution across timescales, due to the complexity of the underlying evolution equations, the nature of multiscale spatio-temporal interactions, and the need to reach long-time integration. We develop a new method that blends numerical solvers with neural operators to accelerate such simulations. This methodology is based on the integration of a community numerical solver with a U-Net neural operator, enhanced by a temporal-conditioning mechanism that enables accurate extrapolation and efficient time-to-solution predictions of the dynamics. We demonstrate the effectiveness of this framework on simulations of microstructure evolution during physical vapor deposition modeled via the phase-field method. Such simulations exhibit high spatial gradients due to the co-evolution of different material phases with simultaneous slow and fast materials dynamics. We establish accurate extrapolation of the coupled solver with up to 16.5$\times$ speed-up compared to DNS. This methodology is generalizable to a broad range of evolutionary models, from solid mechanics, to fluid dynamics, geophysics, climate, and more. △ Less

Submitted 8 December, 2023; originally announced December 2023.

Uncertainty quantification for noisy inputs-outputs in physics-informed neural networks and neural operators

Authors: Zongren Zou, Xuhui Meng, George Em Karniadakis

Abstract: Uncertainty quantification (UQ) in scientific machine learning (SciML) becomes increasingly critical as neural networks (NNs) are being widely adopted in addressing complex problems across various scientific disciplines. Representative SciML models are physics-informed neural networks (PINNs) and neural operators (NOs). While UQ in SciML has been increasingly investigated in recent years, very few… ▽ More Uncertainty quantification (UQ) in scientific machine learning (SciML) becomes increasingly critical as neural networks (NNs) are being widely adopted in addressing complex problems across various scientific disciplines. Representative SciML models are physics-informed neural networks (PINNs) and neural operators (NOs). While UQ in SciML has been increasingly investigated in recent years, very few works have focused on addressing the uncertainty caused by the noisy inputs, such as spatial-temporal coordinates in PINNs and input functions in NOs. The presence of noise in the inputs of the models can pose significantly more challenges compared to noise in the outputs of the models, primarily due to the inherent nonlinearity of most SciML algorithms. As a result, UQ for noisy inputs becomes a crucial factor for reliable and trustworthy deployment of these models in applications involving physical knowledge. To this end, we introduce a Bayesian approach to quantify uncertainty arising from noisy inputs-outputs in PINNs and NOs. We show that this approach can be seamlessly integrated into PINNs and NOs, when they are employed to encode the physical information. PINNs incorporate physics by including physics-informed terms via automatic differentiation, either in the loss function or the likelihood, and often take as input the spatial-temporal coordinate. Therefore, the present method equips PINNs with the capability to address problems where the observed coordinate is subject to noise. On the other hand, pretrained NOs are also commonly employed as equation-free surrogates in solving differential equations and Bayesian inverse problems, in which they take functions as inputs. The proposed approach enables them to handle noisy measurements for both input and output functions with UQ. △ Less

Submitted 19 November, 2023; originally announced November 2023.

Correcting model misspecification in physics-informed neural networks (PINNs)

Authors: Zongren Zou, Xuhui Meng, George Em Karniadakis

Abstract: Data-driven discovery of governing equations in computational science has emerged as a new paradigm for obtaining accurate physical models and as a possible alternative to theoretical derivations. The recently developed physics-informed neural networks (PINNs) have also been employed to learn governing equations given data across diverse scientific disciplines. Despite the effectiveness of PINNs f… ▽ More Data-driven discovery of governing equations in computational science has emerged as a new paradigm for obtaining accurate physical models and as a possible alternative to theoretical derivations. The recently developed physics-informed neural networks (PINNs) have also been employed to learn governing equations given data across diverse scientific disciplines. Despite the effectiveness of PINNs for discovering governing equations, the physical models encoded in PINNs may be misspecified in complex systems as some of the physical processes may not be fully understood, leading to the poor accuracy of PINN predictions. In this work, we present a general approach to correct the misspecified physical models in PINNs for discovering governing equations, given some sparse and/or noisy data. Specifically, we first encode the assumed physical models, which may be misspecified, then employ other deep neural networks (DNNs) to model the discrepancy between the imperfect models and the observational data. Due to the expressivity of DNNs, the proposed method is capable of reducing the computational errors caused by the model misspecification and thus enables the applications of PINNs in complex systems where the physical processes are not exactly known. Furthermore, we utilize the Bayesian PINNs (B-PINNs) and/or ensemble PINNs to quantify uncertainties arising from noisy and/or gappy data in the discovered governing equations. A series of numerical examples including non-Newtonian channel and cavity flows demonstrate that the added DNNs are capable of correcting the model misspecification in PINNs and thus reduce the discrepancy between the physical models and the observational data. We envision that the proposed approach will extend the applications of PINNs for discovering governing equations in problems where the physico-chemical or biological processes are not well understood. △ Less

Submitted 16 October, 2023; originally announced October 2023.

Solution multiplicity and effects of data and eddy viscosity on Navier-Stokes solutions inferred by physics-informed neural networks

Authors: Zhicheng Wang, Xuhui Meng, Xiaomo Jiang, Hui Xiang, George Em Karniadakis

Abstract: Physics-informed neural networks (PINNs) have emerged as a new simulation paradigm for fluid flows and are especially effective for inverse and hybrid problems. However, vanilla PINNs often fail in forward problems, especially at high Reynolds (Re) number flows. Herein, we study systematically the classical lid-driven cavity flow at $Re=2,000$, $3,000$ and $5,000$. We observe that vanilla PINNs ob… ▽ More Physics-informed neural networks (PINNs) have emerged as a new simulation paradigm for fluid flows and are especially effective for inverse and hybrid problems. However, vanilla PINNs often fail in forward problems, especially at high Reynolds (Re) number flows. Herein, we study systematically the classical lid-driven cavity flow at $Re=2,000$, $3,000$ and $5,000$. We observe that vanilla PINNs obtain two classes of solutions, one class that agrees with direct numerical simulations (DNS), and another that is an unstable solution to the Navier-Stokes equations and not physically realizable. We attribute this solution multiplicity to singularities and unbounded vorticity, and we propose regularization methods that restore a unique solution within 1\% difference from the DNS solution. In particular, we introduce a parameterized entropy-viscosity method as artificial eddy viscosity and identify suitable parameters that drive the PINNs solution towards the DNS solution. Furthermore, we solve the inverse problem by subsampling the DNS solution, and identify a new eddy viscosity distribution that leads to velocity and pressure fields almost identical to their DNS counterparts. Surprisingly, a single measurement at a random point suffices to obtain a unique PINNs DNS-like solution even without artificial viscosity, which suggests possible pathways in simulating high Reynolds number turbulent flows using vanilla PINNs. △ Less

Submitted 12 September, 2023; originally announced September 2023.

Characterization of partial wetting by CMAS droplets using multiphase many-body dissipative particle dynamics and data-driven discovery based on PINNs

Authors: Elham Kiyani, Mahdi Kooshkbaghi, Khemraj Shukla, Rahul Babu Koneru, Zhen Li, Luis Bravo, Anindya Ghoshal, George Em Karniadakis, Mikko Karttunen

Abstract: The molten sand, a mixture of calcia, magnesia, alumina, and silicate, known as CMAS, is characterized by its high viscosity, density, and surface tension. The unique properties of CMAS make it a challenging material to deal with in high-temperature applications, requiring innovative solutions and materials to prevent its buildup and damage to critical equipment. Here, we use multiphase many-body… ▽ More The molten sand, a mixture of calcia, magnesia, alumina, and silicate, known as CMAS, is characterized by its high viscosity, density, and surface tension. The unique properties of CMAS make it a challenging material to deal with in high-temperature applications, requiring innovative solutions and materials to prevent its buildup and damage to critical equipment. Here, we use multiphase many-body dissipative particle dynamics (mDPD) simulations to study the wetting dynamics of highly viscous molten CMAS droplets. The simulations are performed in three dimensions, with varying initial droplet sizes and equilibrium contact angles. We propose a coarse parametric ordinary differential equation (ODE) that captures the spreading radius behavior of the CMAS droplets. The ODE parameters are then identified based on the Physics-Informed Neural Network (PINN) framework. Subsequently, the closed form dependency of parameter values found by PINN on the initial radii and contact angles are given using symbolic regression. Finally, we employ Bayesian PINNs (B-PINNs) to assess and quantify the uncertainty associated with the discovered parameters. In brief, this study provides insight into spreading dynamics of CMAS droplets by fusing simple parametric ODE modeling and state-of-the-art machine learning techniques. △ Less

Submitted 18 July, 2023; originally announced July 2023.

Residual-based attention and connection to information bottleneck theory in PINNs

Authors: Sokratis J. Anagnostopoulos, Juan Diego Toscano, Nikolaos Stergiopulos, George Em Karniadakis

Abstract: Driven by the need for more efficient and seamless integration of physical models and data, physics-informed neural networks (PINNs) have seen a surge of interest in recent years. However, ensuring the reliability of their convergence and accuracy remains a challenge. In this work, we propose an efficient, gradient-less weighting scheme for PINNs, that accelerates the convergence of dynamic or sta… ▽ More Driven by the need for more efficient and seamless integration of physical models and data, physics-informed neural networks (PINNs) have seen a surge of interest in recent years. However, ensuring the reliability of their convergence and accuracy remains a challenge. In this work, we propose an efficient, gradient-less weighting scheme for PINNs, that accelerates the convergence of dynamic or static systems. This simple yet effective attention mechanism is a function of the evolving cumulative residuals and aims to make the optimizer aware of problematic regions at no extra computational cost or adversarial learning. We illustrate that this general method consistently achieves a relative $L^{2}$ error of the order of $10^{-5}$ using standard optimizers on typical benchmark cases of the literature. Furthermore, by investigating the evolution of weights during training, we identify two distinct learning phases reminiscent of the fitting and diffusion phases proposed by the information bottleneck (IB) theory. Subsequent gradient analysis supports this hypothesis by aligning the transition from high to low signal-to-noise ratio (SNR) with the transition from fitting to diffusion regimes of the adopted weights. This novel correlation between PINNs and IB theory could open future possibilities for understanding the underlying mechanisms behind the training and stability of PINNs and, more broadly, of neural operators. △ Less

Submitted 1 July, 2023; originally announced July 2023.

MyCrunchGPT: A chatGPT assisted framework for scientific machine learning

Authors: Varun Kumar, Leonard Gleyzer, Adar Kahana, Khemraj Shukla, George Em Karniadakis

Abstract: Scientific Machine Learning (SciML) has advanced recently across many different areas in computational science and engineering. The objective is to integrate data and physics seamlessly without the need of employing elaborate and computationally taxing data assimilation schemes. However, preprocessing, problem formulation, code generation, postprocessing and analysis are still time consuming and m… ▽ More Scientific Machine Learning (SciML) has advanced recently across many different areas in computational science and engineering. The objective is to integrate data and physics seamlessly without the need of employing elaborate and computationally taxing data assimilation schemes. However, preprocessing, problem formulation, code generation, postprocessing and analysis are still time consuming and may prevent SciML from wide applicability in industrial applications and in digital twin frameworks. Here, we integrate the various stages of SciML under the umbrella of ChatGPT, to formulate MyCrunchGPT, which plays the role of a conductor orchestrating the entire workflow of SciML based on simple prompts by the user. Specifically, we present two examples that demonstrate the potential use of MyCrunchGPT in optimizing airfoils in aerodynamics, and in obtaining flow fields in various geometries in interactive mode, with emphasis on the validation stage. To demonstrate the flow of the MyCrunchGPT, and create an infrastructure that can facilitate a broader vision, we built a webapp based guided user interface, that includes options for a comprehensive summary report. The overall objective is to extend MyCrunchGPT to handle diverse problems in computational mechanics, design, optimization and controls, and general scientific computing tasks involved in SciML, hence using it as a research assistant tool but also as an educational tool. While here the examples focus in fluid mechanics, future versions will target solid mechanics and materials science, geophysics, systems biology and bioinformatics. △ Less

Submitted 31 July, 2023; v1 submitted 27 June, 2023; originally announced June 2023.

Comments: Updated title, abstract and added references

Dynamic spreading and infiltration of a molten sand droplet on a porous surface

Authors: Rahul Babu Koneru, Garrett Foresman, Alison Flatau, Zhen Li, Luis Bravo, Muthuvel Murugan, Anindya Ghoshal, George Em Karniadakis

Abstract: Compared to smooth surfaces, droplet spreading on porous surfaces is more complex and has relevance in many engineering applications. In this work, we investigate the infiltration dynamics of molten sand droplets on structured porous surfaces using the multiphase many-body dissipative particle dynamics (mDPD) method. We carry out three-dimensional simulations with different equilibrium contact ang… ▽ More Compared to smooth surfaces, droplet spreading on porous surfaces is more complex and has relevance in many engineering applications. In this work, we investigate the infiltration dynamics of molten sand droplets on structured porous surfaces using the multiphase many-body dissipative particle dynamics (mDPD) method. We carry out three-dimensional simulations with different equilibrium contact angles and surface porosities. The temporal evolution of the radius of the wetted area follows a power law, as in the case of a smooth surface. The infiltration rate on the other hand is dictated by the competition between spreading and capillary inhibition of the pores. Additionally, the temporal evolution of the droplet height and the contact angle on the porous surface is also presented. △ Less

Submitted 6 March, 2023; originally announced March 2023.

A unified scalable framework for causal sweeping strategies for Physics-Informed Neural Networks (PINNs) and their temporal decompositions

Authors: Michael Penwarden, Ameya D. Jagtap, Shandian Zhe, George Em Karniadakis, Robert M. Kirby

Abstract: Physics-informed neural networks (PINNs) as a means of solving partial differential equations (PDE) have garnered much attention in the Computational Science and Engineering (CS&E) world. However, a recent topic of interest is exploring various training (i.e., optimization) challenges - in particular, arriving at poor local minima in the optimization landscape results in a PINN approximation givin… ▽ More Physics-informed neural networks (PINNs) as a means of solving partial differential equations (PDE) have garnered much attention in the Computational Science and Engineering (CS&E) world. However, a recent topic of interest is exploring various training (i.e., optimization) challenges - in particular, arriving at poor local minima in the optimization landscape results in a PINN approximation giving an inferior, and sometimes trivial, solution when solving forward time-dependent PDEs with no data. This problem is also found in, and in some sense more difficult, with domain decomposition strategies such as temporal decomposition using XPINNs. We furnish examples and explanations for different training challenges, their cause, and how they relate to information propagation and temporal decomposition. We then propose a new stacked-decomposition method that bridges the gap between time-marching PINNs and XPINNs. We also introduce significant computational speed-ups by using transfer learning concepts to initialize subnetworks in the domain and loss tolerance-based propagation for the subdomains. Finally, we formulate a new time-sweeping collocation point algorithm inspired by the previous PINNs causality literature, which our framework can still describe, and provides a significant computational speed-up via reduced-cost collocation point segmentation. The proposed methods form our unified framework, which overcomes training challenges in PINNs and XPINNs for time-dependent PDEs by respecting the causality in multiple forms and improving scalability by limiting the computation required per optimization iteration. Finally, we provide numerical results for these methods on baseline PDE problems for which unmodified PINNs and XPINNs struggle to train. △ Less

Submitted 18 September, 2023; v1 submitted 27 February, 2023; originally announced February 2023.

Journal ref: Journal of Computational Physics, 493, 2023, 112464

Learning stiff chemical kinetics using extended deep neural operators

Authors: Somdatta Goswami, Ameya D. Jagtap, Hessam Babaee, Bryan T. Susi, George Em Karniadakis

Abstract: We utilize neural operators to learn the solution propagator for the challenging chemical kinetics equation. Specifically, we apply the deep operator network (DeepONet) along with its extensions, such as the autoencoder-based DeepONet and the newly proposed Partition-of-Unity (PoU-) DeepONet to study a range of examples, including the ROBERS problem with three species, the POLLU problem with 25 sp… ▽ More We utilize neural operators to learn the solution propagator for the challenging chemical kinetics equation. Specifically, we apply the deep operator network (DeepONet) along with its extensions, such as the autoencoder-based DeepONet and the newly proposed Partition-of-Unity (PoU-) DeepONet to study a range of examples, including the ROBERS problem with three species, the POLLU problem with 25 species, pure kinetics of the syngas skeletal model for $CO/H_2$ burning, which contains 11 species and 21 reactions and finally, a temporally developing planar $CO/H_2$ jet flame (turbulent flame) using the same syngas mechanism. We have demonstrated the advantages of the proposed approach through these numerical examples. Specifically, to train the DeepONet for the syngas model, we solve the skeletal kinetic model for different initial conditions. In the first case, we parametrize the initial conditions based on equivalence ratios and initial temperature values. In the second case, we perform a direct numerical simulation of a two-dimensional temporally developing $CO/H_2$ jet flame. Then, we initialize the kinetic model by the thermochemical states visited by a subset of grid points at different time snapshots. Stiff problems are computationally expensive to solve with traditional stiff solvers. Thus, this work aims to develop a neural operator-based surrogate model to solve stiff chemical kinetics. The operator, once trained offline, can accurately integrate the thermochemical state for arbitrarily large time advancements, leading to significant computational gains compared to stiff integration schemes. △ Less

Submitted 23 February, 2023; originally announced February 2023.

Comments: 21 pages, 11 figures

Deep neural operators can predict the real-time response of floating offshore structures under irregular waves

Authors: Qianying Cao, Somdatta Goswami, Tapas Tripura, Souvik Chakraborty, George Em Karniadakis

Abstract: The use of neural operators in a digital twin model of an offshore floating structure can provide a paradigm shift in structural response prediction and health monitoring, providing valuable information for real-time control. In this work, the performance of three neural operators is evaluated, namely, deep operator network (DeepONet), Fourier neural operator (FNO), and Wavelet neural operator (WN… ▽ More The use of neural operators in a digital twin model of an offshore floating structure can provide a paradigm shift in structural response prediction and health monitoring, providing valuable information for real-time control. In this work, the performance of three neural operators is evaluated, namely, deep operator network (DeepONet), Fourier neural operator (FNO), and Wavelet neural operator (WNO). We investigate the effectiveness of the operators to accurately capture the responses of a floating structure under six different sea state codes $(3-8)$ based on the wave characteristics described by the World Meteorological Organization (WMO). The results demonstrate that these high-precision neural operators can deliver structural responses more efficiently, up to two orders of magnitude faster than a dynamic analysis using conventional numerical solvers. Additionally, compared to gated recurrent units (GRUs), a commonly used recurrent neural network for time-series estimation, neural operators are both more accurate and efficient, especially in situations with limited data availability. To further enhance the accuracy, novel extensions, such as wavelet-DeepONet and self-adaptive WNO, are proposed. Taken together, our study shows that FNO outperforms all other operators for approximating the mapping of one input functional space to the output space as well as for responses that have small bandwidth of the frequency spectrum, whereas for learning the mapping of multiple functions in the input space to the output space as well as for capturing responses within a large frequency spectrum, DeepONet with historical states provides the highest accuracy. △ Less

Submitted 30 November, 2023; v1 submitted 13 February, 2023; originally announced February 2023.

Comments: 40 pages, 16 figures, 13 tables

Learning bias corrections for climate models using deep neural operators

Authors: Aniruddha Bora, Khemraj Shukla, Shixuan Zhang, Bryce Harrop, Ruby Leung, George Em Karniadakis

Abstract: Numerical simulation for climate modeling resolving all important scales is a computationally taxing process. Therefore, to circumvent this issue a low resolution simulation is performed, which is subsequently corrected for bias using reanalyzed data (ERA5), known as nudging correction. The existing implementation for nudging correction uses a relaxation based method for the algebraic difference b… ▽ More Numerical simulation for climate modeling resolving all important scales is a computationally taxing process. Therefore, to circumvent this issue a low resolution simulation is performed, which is subsequently corrected for bias using reanalyzed data (ERA5), known as nudging correction. The existing implementation for nudging correction uses a relaxation based method for the algebraic difference between low resolution and ERA5 data. In this study, we replace the bias correction process with a surrogate model based on the Deep Operator Network (DeepONet). DeepONet (Deep Operator Neural Network) learns the mapping from the state before nudging (a functional) to the nudging tendency (another functional). The nudging tendency is a very high dimensional data albeit having many low energy modes. Therefore, the DeepoNet is combined with a convolution based auto-encoder-decoder (AED) architecture in order to learn the nudging tendency in a lower dimensional latent space efficiently. The accuracy of the DeepONet model is tested against the nudging tendency obtained from the E3SMv2 (Energy Exascale Earth System Model) and shows good agreement. The overarching goal of this work is to deploy the DeepONet model in an online setting and replace the nudging module in the E3SM loop for better efficiency and accuracy. △ Less

Submitted 6 February, 2023; originally announced February 2023.

Deep neural operators can serve as accurate surrogates for shape optimization: A case study for airfoils

Authors: Khemraj Shukla, Vivek Oommen, Ahmad Peyvan, Michael Penwarden, Luis Bravo, Anindya Ghoshal, Robert M. Kirby, George Em Karniadakis

Abstract: Deep neural operators, such as DeepONets, have changed the paradigm in high-dimensional nonlinear regression from function regression to (differential) operator regression, paving the way for significant changes in computational engineering applications. Here, we investigate the use of DeepONets to infer flow fields around unseen airfoils with the aim of shape optimization, an important design pro… ▽ More Deep neural operators, such as DeepONets, have changed the paradigm in high-dimensional nonlinear regression from function regression to (differential) operator regression, paving the way for significant changes in computational engineering applications. Here, we investigate the use of DeepONets to infer flow fields around unseen airfoils with the aim of shape optimization, an important design problem in aerodynamics that typically taxes computational resources heavily. We present results which display little to no degradation in prediction accuracy, while reducing the online optimization cost by orders of magnitude. We consider NACA airfoils as a test case for our proposed approach, as their shape can be easily defined by the four-digit parametrization. We successfully optimize the constrained NACA four-digit problem with respect to maximizing the lift-to-drag ratio and validate all results by comparing them to a high-order CFD solver. We find that DeepONets have low generalization error, making them ideal for generating solutions of unseen shapes. Specifically, pressure, density, and velocity fields are accurately inferred at a fraction of a second, hence enabling the use of general objective functions beyond the maximization of the lift-to-drag ratio considered in the current work. △ Less

Submitted 1 February, 2023; originally announced February 2023.

Comments: 21 pages, 14 Figures

A Hybrid Deep Neural Operator/Finite Element Method for Ice-Sheet Modeling

Authors: QiZhi He, Mauro Perego, Amanda A. Howard, George Em Karniadakis, Panos Stinis

Abstract: One of the most challenging and consequential problems in climate modeling is to provide probabilistic projections of sea level rise. A large part of the uncertainty of sea level projections is due to uncertainty in ice sheet dynamics. At the moment, accurate quantification of the uncertainty is hindered by the cost of ice sheet computational models. In this work, we develop a hybrid approach to a… ▽ More One of the most challenging and consequential problems in climate modeling is to provide probabilistic projections of sea level rise. A large part of the uncertainty of sea level projections is due to uncertainty in ice sheet dynamics. At the moment, accurate quantification of the uncertainty is hindered by the cost of ice sheet computational models. In this work, we develop a hybrid approach to approximate existing ice sheet computational models at a fraction of their cost. Our approach consists of replacing the finite element model for the momentum equations for the ice velocity, the most expensive part of an ice sheet model, with a Deep Operator Network, while retaining a classic finite element discretization for the evolution of the ice thickness. We show that the resulting hybrid model is very accurate and it is an order of magnitude faster than the traditional finite element model. Further, a distinctive feature of the proposed model compared to other neural network approaches, is that it can handle high-dimensional parameter spaces (parameter fields) such as the basal friction at the bed of the glacier, and can therefore be used for generating samples for uncertainty quantification. We study the impact of hyper-parameters, number of unknowns and correlation length of the parameter distribution on the training and accuracy of the Deep Operator Network on a synthetic ice sheet model. We then target the evolution of the Humboldt glacier in Greenland and show that our hybrid model can provide accurate statistics of the glacier mass loss and can be effectively used to accelerate the quantification of uncertainty. △ Less

Submitted 26 January, 2023; originally announced January 2023.

L-HYDRA: Multi-Head Physics-Informed Neural Networks

Authors: Zongren Zou, George Em Karniadakis

Abstract: We introduce multi-head neural networks (MH-NNs) to physics-informed machine learning, which is a type of neural networks (NNs) with all nonlinear hidden layers as the body and multiple linear output layers as multi-head. Hence, we construct multi-head physics-informed neural networks (MH-PINNs) as a potent tool for multi-task learning (MTL), generative modeling, and few-shot learning for diverse… ▽ More We introduce multi-head neural networks (MH-NNs) to physics-informed machine learning, which is a type of neural networks (NNs) with all nonlinear hidden layers as the body and multiple linear output layers as multi-head. Hence, we construct multi-head physics-informed neural networks (MH-PINNs) as a potent tool for multi-task learning (MTL), generative modeling, and few-shot learning for diverse problems in scientific machine learning (SciML). MH-PINNs connect multiple functions/tasks via a shared body as the basis functions as well as a shared distribution for the head. The former is accomplished by solving multiple tasks with MH-PINNs with each head independently corresponding to each task, while the latter by employing normalizing flows (NFs) for density estimate and generative modeling. To this end, our method is a two-stage method, and both stages can be tackled with standard deep learning tools of NNs, enabling easy implementation in practice. MH-PINNs can be used for various purposes, such as approximating stochastic processes, solving multiple tasks synergistically, providing informative prior knowledge for downstream few-shot learning tasks such as meta-learning and transfer learning, learning representative basis functions, and uncertainty quantification. We demonstrate the effectiveness of MH-PINNs in five benchmarks, investigating also the possibility of synergistic learning in regression analysis. We name the open-source code "Lernaean Hydra" (L-HYDRA), since this mythical creature possessed many heads for performing important multiple tasks, as in the proposed method. △ Less

Submitted 5 January, 2023; originally announced January 2023.

MSC Class: 34F05; 62M45; 65L99; 65M99; 65N99

Reliable extrapolation of deep neural operators informed by physics or sparse observations

Authors: Min Zhu, Handi Zhang, Anran Jiao, George Em Karniadakis, Lu Lu

Abstract: Deep neural operators can learn nonlinear mappings between infinite-dimensional function spaces via deep neural networks. As promising surrogate solvers of partial differential equations (PDEs) for real-time prediction, deep neural operators such as deep operator networks (DeepONets) provide a new simulation paradigm in science and engineering. Pure data-driven neural operators and deep learning m… ▽ More Deep neural operators can learn nonlinear mappings between infinite-dimensional function spaces via deep neural networks. As promising surrogate solvers of partial differential equations (PDEs) for real-time prediction, deep neural operators such as deep operator networks (DeepONets) provide a new simulation paradigm in science and engineering. Pure data-driven neural operators and deep learning models, in general, are usually limited to interpolation scenarios, where new predictions utilize inputs within the support of the training set. However, in the inference stage of real-world applications, the input may lie outside the support, i.e., extrapolation is required, which may result to large errors and unavoidable failure of deep learning models. Here, we address this challenge of extrapolation for deep neural operators. First, we systematically investigate the extrapolation behavior of DeepONets by quantifying the extrapolation complexity via the 2-Wasserstein distance between two function spaces and propose a new behavior of bias-variance trade-off for extrapolation with respect to model capacity. Subsequently, we develop a complete workflow, including extrapolation determination, and we propose five reliable learning methods that guarantee a safe prediction under extrapolation by requiring additional information -- the governing PDEs of the system or sparse new observations. The proposed methods are based on either fine-tuning a pre-trained DeepONet or multifidelity learning. We demonstrate the effectiveness of the proposed framework for various types of parametric PDEs. Our systematic comparisons provide practical guidelines for selecting a proper extrapolation method depending on the available information, desired accuracy, and required inference speed. △ Less

Submitted 12 December, 2022; originally announced December 2022.

NeuralUQ: A comprehensive library for uncertainty quantification in neural differential equations and operators

Authors: Zongren Zou, Xuhui Meng, Apostolos F Psaros, George Em Karniadakis

Abstract: Uncertainty quantification (UQ) in machine learning is currently drawing increasing research interest, driven by the rapid deployment of deep neural networks across different fields, such as computer vision, natural language processing, and the need for reliable tools in risk-sensitive applications. Recently, various machine learning models have also been developed to tackle problems in the field… ▽ More Uncertainty quantification (UQ) in machine learning is currently drawing increasing research interest, driven by the rapid deployment of deep neural networks across different fields, such as computer vision, natural language processing, and the need for reliable tools in risk-sensitive applications. Recently, various machine learning models have also been developed to tackle problems in the field of scientific computing with applications to computational science and engineering (CSE). Physics-informed neural networks and deep operator networks are two such models for solving partial differential equations and learning operator mappings, respectively. In this regard, a comprehensive study of UQ methods tailored specifically for scientific machine learning (SciML) models has been provided in [45]. Nevertheless, and despite their theoretical merit, implementations of these methods are not straightforward, especially in large-scale CSE applications, hindering their broad adoption in both research and industry settings. In this paper, we present an open-source Python library (https://github.com/Crunch-UQ4MI), termed NeuralUQ and accompanied by an educational tutorial, for employing UQ methods for SciML in a convenient and structured manner. The library, designed for both educational and research purposes, supports multiple modern UQ methods and SciML models. It is based on a succinct workflow and facilitates flexible employment and easy extensions by the users. We first present a tutorial of NeuralUQ and subsequently demonstrate its applicability and efficiency in four diverse examples, involving dynamical systems and high-dimensional parametric and time-dependent PDEs. △ Less

Submitted 25 August, 2022; originally announced August 2022.

Comments: 27 pages, 12 figures

Fractional SEIR Model and Data-Driven Predictions of COVID-19 Dynamics of Omicron Variant

Authors: Min Cai, George Em Karniadakis, Changpin Li

Abstract: We study the dynamic evolution of COVID-19 cased by the Omicron variant via a fractional susceptible-exposedinfected-removed (SEIR) model. Preliminary data suggest that the symptoms of Omicron infection are not prominent and the transmission is therefore more concealed, which causes a relatively slow increase in the detected cases of the new infected at the beginning of the pandemic. To characteri… ▽ More We study the dynamic evolution of COVID-19 cased by the Omicron variant via a fractional susceptible-exposedinfected-removed (SEIR) model. Preliminary data suggest that the symptoms of Omicron infection are not prominent and the transmission is therefore more concealed, which causes a relatively slow increase in the detected cases of the new infected at the beginning of the pandemic. To characterize the specific dynamics, the Caputo-Hadamard fractional derivative is adopted to refined the classical SEIR model. Based on the reported data, we infer the fractional order, timedependent parameters, as well as unobserved dynamics of the fractional SEIR model via fractional physics-informed neural networks (fPINNs). Then, we make short-time predictions using the learned fractional SEIR model. △ Less

Submitted 23 May, 2022; originally announced May 2022.

Journal ref: Chaos 32, 071101 (2022)

Learning two-phase microstructure evolution using neural operators and autoencoder architectures

Authors: Vivek Oommen, Khemraj Shukla, Somdatta Goswami, Remi Dingreville, George Em Karniadakis

Abstract: Phase-field modeling is an effective but computationally expensive method for capturing the mesoscale morphological and microstructure evolution in materials. Hence, fast and generalizable surrogate models are needed to alleviate the cost of computationally taxing processes such as in optimization and design of materials. The intrinsic discontinuous nature of the physical phenomena incurred by the… ▽ More Phase-field modeling is an effective but computationally expensive method for capturing the mesoscale morphological and microstructure evolution in materials. Hence, fast and generalizable surrogate models are needed to alleviate the cost of computationally taxing processes such as in optimization and design of materials. The intrinsic discontinuous nature of the physical phenomena incurred by the presence of sharp phase boundaries makes the training of the surrogate model cumbersome. We develop a framework that integrates a convolutional autoencoder architecture with a deep neural operator (DeepONet) to learn the dynamic evolution of a two-phase mixture and accelerate time-to-solution in predicting the microstructure evolution. We utilize the convolutional autoencoder to provide a compact representation of the microstructure data in a low-dimensional latent space. DeepONet, which consists of two sub-networks, one for encoding the input function at a fixed number of sensors locations (branch net) and another for encoding the locations for the output functions (trunk net), learns the mesoscale dynamics of the microstructure evolution from the autoencoder latent space. The decoder part of the convolutional autoencoder then reconstructs the time-evolved microstructure from the DeepONet predictions. The trained DeepONet architecture can then be used to replace the high-fidelity phase-field numerical solver in interpolation tasks or to accelerate the numerical solver in extrapolation tasks. △ Less

Submitted 29 June, 2022; v1 submitted 11 April, 2022; originally announced April 2022.

Discovering and forecasting extreme events via active learning in neural operators

Authors: Ethan Pickering, Stephen Guth, George Em Karniadakis, Themistoklis P. Sapsis

Abstract: Extreme events in society and nature, such as pandemic spikes, rogue waves, or structural failures, can have catastrophic consequences. Characterizing extremes is difficult as they occur rarely, arise from seemingly benign conditions, and belong to complex and often unknown infinite-dimensional systems. Such challenges render attempts at characterizing them as moot. We address each of these diffic… ▽ More Extreme events in society and nature, such as pandemic spikes, rogue waves, or structural failures, can have catastrophic consequences. Characterizing extremes is difficult as they occur rarely, arise from seemingly benign conditions, and belong to complex and often unknown infinite-dimensional systems. Such challenges render attempts at characterizing them as moot. We address each of these difficulties by combining novel training schemes in Bayesian experimental design (BED) with an ensemble of deep neural operators (DNOs). This model-agnostic framework pairs a BED scheme that actively selects data for quantifying extreme events with an ensemble of DNOs that approximate infinite-dimensional nonlinear operators. We find that not only does this framework clearly beat Gaussian processes (GPs) but that 1) shallow ensembles of just two members perform best; 2) extremes are uncovered regardless of the state of initial data (i.e. with or without extremes); 3) our method eliminates "double-descent" phenomena; 4) the use of batches of suboptimal acquisition points compared to step-by-step global optima does not hinder BED performance; and 5) Monte Carlo acquisition outperforms standard optimizers in high-dimensions. Together these conclusions form the foundation of an AI-assisted experimental infrastructure that can efficiently infer and pinpoint critical situations across many domains, from physical to societal systems. △ Less

Submitted 20 September, 2022; v1 submitted 5 April, 2022; originally announced April 2022.

Comments: 25 pages, 8 figures, Submitted to Nature Computational Science

Deep learning of inverse water waves problems using multi-fidelity data: Application to Serre-Green-Naghdi equations

Authors: Ameya D. Jagtap, Dimitrios Mitsotakis, George Em Karniadakis

Abstract: We consider strongly-nonlinear and weakly-dispersive surface water waves governed by equations of Boussinesq type, known as the Serre-Green-Naghdi system; it describes future states of the free water surface and depth averaged horizontal velocity, given their initial state. The lack of knowledge of the velocity field as well as the initial states provided by measurements lead to an ill-posed probl… ▽ More We consider strongly-nonlinear and weakly-dispersive surface water waves governed by equations of Boussinesq type, known as the Serre-Green-Naghdi system; it describes future states of the free water surface and depth averaged horizontal velocity, given their initial state. The lack of knowledge of the velocity field as well as the initial states provided by measurements lead to an ill-posed problem that cannot be solved by traditional techniques. To this end, we employ physics-informed neural networks (PINNs) to generate solutions to such ill-posed problems using only data of the free surface elevation and depth of the water. PINNs can readily incorporate the physical laws and the observational data, thereby enabling inference of the physical quantities of interest. In the present study, both experimental and synthetic (generated by numerical methods) training data are used to train PINNs. Furthermore, multi-fidelity data are used to solve the inverse water wave problem by leveraging both high- and low-fidelity data sets. The applicability of the PINN methodology for the estimation of the impact of water waves onto solid obstacles is demonstrated after deriving the corresponding equations. The present methodology can be employed to efficiently design offshore structures such as oil platforms, wind turbines, etc. by solving the corresponding ill-posed inverse water waves problem. △ Less

Submitted 6 February, 2022; originally announced February 2022.

A comprehensive and fair comparison of two neural operators (with practical extensions) based on FAIR data

Authors: Lu Lu, Xuhui Meng, Shengze Cai, Zhiping Mao, Somdatta Goswami, Zhongqiang Zhang, George Em Karniadakis

Abstract: Neural operators can learn nonlinear mappings between function spaces and offer a new simulation paradigm for real-time prediction of complex dynamics for realistic diverse applications as well as for system identification in science and engineering. Herein, we investigate the performance of two neural operators, and we develop new practical extensions that will make them more accurate and robust… ▽ More Neural operators can learn nonlinear mappings between function spaces and offer a new simulation paradigm for real-time prediction of complex dynamics for realistic diverse applications as well as for system identification in science and engineering. Herein, we investigate the performance of two neural operators, and we develop new practical extensions that will make them more accurate and robust and importantly more suitable for industrial-complexity applications. The first neural operator, DeepONet, was published in 2019, and the second one, named Fourier Neural Operator or FNO, was published in 2020. In order to compare FNO with DeepONet for realistic setups, we develop several extensions of FNO that can deal with complex geometric domains as well as mappings where the input and output function spaces are of different dimensions. We also endow DeepONet with special features that provide inductive bias and accelerate training, and we present a faster implementation of DeepONet with cost comparable to the computational cost of FNO. We consider 16 different benchmarks to demonstrate the relative performance of the two neural operators, including instability wave analysis in hypersonic boundary layers, prediction of the vorticity field of a flapping airfoil, porous media simulations in complex-geometry domains, etc. The performance of DeepONet and FNO is comparable for relatively simple settings, but for complex geometries and especially noisy data, the performance of FNO deteriorates greatly. For example, for the instability wave analysis with only 0.1% noise added to the input data, the error of FNO increases 10000 times making it inappropriate for such important applications, while there is hardly any effect of such noise on the DeepONet. We also compare theoretically the two neural operators and obtain similar error estimates for DeepONet and FNO under the same regularity assumptions. △ Less

Submitted 9 November, 2021; originally announced November 2021.

Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems

Authors: Jeremy Yu, Lu Lu, Xuhui Meng, George Em Karniadakis

Abstract: Deep learning has been shown to be an effective tool in solving partial differential equations (PDEs) through physics-informed neural networks (PINNs). PINNs embed the PDE residual into the loss function of the neural network, and have been successfully employed to solve diverse forward and inverse PDE problems. However, one disadvantage of the first generation of PINNs is that they usually have l… ▽ More Deep learning has been shown to be an effective tool in solving partial differential equations (PDEs) through physics-informed neural networks (PINNs). PINNs embed the PDE residual into the loss function of the neural network, and have been successfully employed to solve diverse forward and inverse PDE problems. However, one disadvantage of the first generation of PINNs is that they usually have limited accuracy even with many training points. Here, we propose a new method, gradient-enhanced physics-informed neural networks (gPINNs), for improving the accuracy and training efficiency of PINNs. gPINNs leverage gradient information of the PDE residual and embed the gradient into the loss function. We tested gPINNs extensively and demonstrated the effectiveness of gPINNs in both forward and inverse PDE problems. Our numerical results show that gPINN performs better than PINN with fewer training points. Furthermore, we combined gPINN with the method of residual-based adaptive refinement (RAR), a method for improving the distribution of training points adaptively during training, to further improve the performance of gPINN, especially in PDEs with solutions that have steep gradients. △ Less

Submitted 1 November, 2021; originally announced November 2021.

Physics-informed Neural Networks (PINNs) for Wave Propagation and Full Waveform Inversions

Authors: Majid Rasht-Behesht, Christian Huber, Khemraj Shukla, George Em Karniadakis

Abstract: We propose a new approach to the solution of the wave propagation and full waveform inversions (FWIs) based on a recent advance in deep learning called Physics-Informed Neural Networks (PINNs). In this study, we present an algorithm for PINNs applied to the 2D acoustic wave equation and test the model with both forward wave propagation and FWIs case studies. These synthetic case studies are design… ▽ More We propose a new approach to the solution of the wave propagation and full waveform inversions (FWIs) based on a recent advance in deep learning called Physics-Informed Neural Networks (PINNs). In this study, we present an algorithm for PINNs applied to the 2D acoustic wave equation and test the model with both forward wave propagation and FWIs case studies. These synthetic case studies are designed to explore the ability of PINNs to handle varying degrees of structural complexity using both teleseismic plane waves and seismic point sources. PINNs meshless formalism allows for a flexible implementation of the wave equation and different types of boundary conditions. For instance, our models demonstrate that PINN automatically satisfies absorbing boundary conditions, a serious computational challenge for common wave propagation solvers. Furthermore, a priori knowledge of the subsurface structure can be seamlessly encoded in PINNs formulation. We find that the current state-of-the-art PINNs provide good results for the forward model, even though spectral element or finite difference methods are more efficient and accurate. More importantly, our results demonstrate that PINNs yield excellent results for inversions on all cases considered and with limited computational complexity. Using PINNs as a geophysical inversion solver offers exciting perspectives, not only for the full waveform seismic inversions, but also when dealing with other geophysical datasets (e.g., magnetotellurics, gravity) as well as joint inversions because of its robust framework and simple implementation. △ Less

Submitted 26 August, 2021; originally announced August 2021.

Physics-informed neural networks (PINNs) for fluid mechanics: A review

Authors: Shengze Cai, Zhiping Mao, Zhicheng Wang, Minglang Yin, George Em Karniadakis

Abstract: Despite the significant progress over the last 50 years in simulating flow problems using numerical discretization of the Navier-Stokes equations (NSE), we still cannot incorporate seamlessly noisy data into existing algorithms, mesh-generation is complex, and we cannot tackle high-dimensional problems governed by parametrized NSE. Moreover, solving inverse flow problems is often prohibitively exp… ▽ More Despite the significant progress over the last 50 years in simulating flow problems using numerical discretization of the Navier-Stokes equations (NSE), we still cannot incorporate seamlessly noisy data into existing algorithms, mesh-generation is complex, and we cannot tackle high-dimensional problems governed by parametrized NSE. Moreover, solving inverse flow problems is often prohibitively expensive and requires complex and expensive formulations and new computer codes. Here, we review flow physics-informed learning, integrating seamlessly data and mathematical models, and implementing them using physics-informed neural networks (PINNs). We demonstrate the effectiveness of PINNs for inverse problems related to three-dimensional wake flows, supersonic flows, and biomedical flows. △ Less

Submitted 20 May, 2021; originally announced May 2021.

A physics-informed neural network for quantifying the microstructure properties of polycrystalline Nickel using ultrasound data

Authors: Khemraj Shukla, Ameya D. Jagtap, James L. Blackshire, Daniel Sparkman, George Em Karniadakis

Abstract: We employ physics-informed neural networks (PINNs) to quantify the microstructure of a polycrystalline Nickel by computing the spatial variation of compliance coefficients (compressibility, stiffness and rigidity) of the material. The PINN is supervised with realistic ultrasonic surface acoustic wavefield data acquired at an ultrasonic frequency of 5 MHz for the polycrystalline material. The ultra… ▽ More We employ physics-informed neural networks (PINNs) to quantify the microstructure of a polycrystalline Nickel by computing the spatial variation of compliance coefficients (compressibility, stiffness and rigidity) of the material. The PINN is supervised with realistic ultrasonic surface acoustic wavefield data acquired at an ultrasonic frequency of 5 MHz for the polycrystalline material. The ultrasonic wavefield data is represented as a deformation on the top surface of the material with the deformation measured using the method of laser vibrometry. The ultrasonic data is further complemented with wavefield data generated using a finite element based solver. The neural network is physically-informed by the in-plane and out-of-plane elastic wave equations and its convergence is accelerated using adaptive activation functions. The overarching goal of this work is to infer the spatial variation of compliance coefficients of materials using PINNs, which for ultrasound involves the spatially varying speed of the elastic waves. More broadly, the resulting PINN based surrogate model shows a promising approach for solving ill-posed inverse problems, often encountered in the non-destructive evaluation of materials. △ Less

Submitted 5 October, 2021; v1 submitted 25 March, 2021; originally announced March 2021.

Comments: 18 pages, 5 figures

Flow over an espresso cup: Inferring 3D velocity and pressure fields from tomographic background oriented schlieren videos via physics-informed neural networks

Authors: Shengze Cai, Zhicheng Wang, Frederik Fuest, Young-Jin Jeon, Callum Gray, George Em Karniadakis

Abstract: Tomographic background oriented schlieren (Tomo-BOS) imaging measures density or temperature fields in 3D using multiple camera BOS projections, and is particularly useful for instantaneous flow visualizations of complex fluid dynamics problems. We propose a new method based on physics-informed neural networks (PINNs) to infer the full continuous 3D velocity and pressure fields from snapshots of 3… ▽ More Tomographic background oriented schlieren (Tomo-BOS) imaging measures density or temperature fields in 3D using multiple camera BOS projections, and is particularly useful for instantaneous flow visualizations of complex fluid dynamics problems. We propose a new method based on physics-informed neural networks (PINNs) to infer the full continuous 3D velocity and pressure fields from snapshots of 3D temperature fields obtained by Tomo-BOS imaging. PINNs seamlessly integrate the underlying physics of the observed fluid flow and the visualization data, hence enabling the inference of latent quantities using limited experimental data. In this hidden fluid mechanics paradigm, we train the neural network by minimizing a loss function composed of a data mismatch term and residual terms associated with the coupled Navier-Stokes and heat transfer equations. We first quantify the accuracy of the proposed method based on a 2D synthetic data set for buoyancy-driven flow, and subsequently apply it to the Tomo-BOS data set, where we are able to infer the instantaneous velocity and pressure fields of the flow over an espresso cup based only on the temperature field provided by the Tomo-BOS imaging. Moreover, we conduct an independent PIV experiment to validate the PINN inference for the unsteady velocity field at a center plane. To explain the observed flow physics, we also perform systematic PINN simulations at different Reynolds and Richardson numbers and quantify the variations in velocity and pressure fields. The results in this paper indicate that the proposed deep learning technique can become a promising direction in experimental fluid mechanics. △ Less

Submitted 9 March, 2021; v1 submitted 3 March, 2021; originally announced March 2021.

Comments: 17 pages, supplementary materials attached

Multiscale Parareal Algorithm for Long-Time Mesoscopic Simulations of Microvascular Blood Flow in Zebrafish

Authors: Ansel Blumers, Minglang Yin, Hiroyuki Nakajima, Yosuke Hasegawa, Zhen Li, George Em Karniadakis

Abstract: Various biological processes such as transport of oxygen and nutrients, thrombus formation, vascular angiogenesis and remodeling are related to cellular/subcellular level biological processes, where mesoscopic simulations resolving detailed cell dynamics provide a key to understanding and identifying the cellular basis of disease. To break this bottleneck and achieve a biologically meaningful time… ▽ More Various biological processes such as transport of oxygen and nutrients, thrombus formation, vascular angiogenesis and remodeling are related to cellular/subcellular level biological processes, where mesoscopic simulations resolving detailed cell dynamics provide a key to understanding and identifying the cellular basis of disease. To break this bottleneck and achieve a biologically meaningful timescale, we propose a multiscale parareal algorithm in which a continuum-based solver supervises a mesoscopic simulation in the time-domain. Using an iterative prediction-correction strategy, the parallel-in-time mesoscopic simulation supervised by its continuum-based counterpart can converge fast. The effectiveness of the proposed method is first verified in a time-dependent flow with a sinusoidal flowrate through a Y-shaped bifurcation channel. Physical quantities of interest including velocity, wall shear stress and flowrate are computed to compare against those of reference solutions, showing a less than 1% relative error on flowrate in the Newtonian flow and a less than 3\% relative error in the non-Newtonian blood flow. The proposed method is then applied to a large-scale mesoscopic simulation of microvessel blood flow in a zebrafish hindbrain for temporal acceleration. The time-dependent blood flow from heartbeats in this realistic vascular network of zebrafish hindbrain is simulated using dissipative particle dynamics as the mesoscopic model, which is supervised by a one-dimensional blood flow model (continuum-based model) in multiple temporal sub-domains. The computational analysis shows that the resulting microvessel blood flow converges to the reference solution after only two iterations. The proposed method is suitable for long-time mesoscopic simulations with complex fluids and geometries. △ Less

Submitted 20 January, 2021; originally announced January 2021.

A fast multi-fidelity method with uncertainty quantification for complex data correlations: Application to vortex-induced vibrations of marine risers

Authors: Xuhui Meng, Zhicheng Wang, Dixia Fan, Michael Triantafyllou, George Em Karniadakis

Abstract: We develop a fast multi-fidelity modeling method for very complex correlations between high- and low-fidelity data by working in modal space to extract the proper correlation function. We apply this method to infer the amplitude of motion of a flexible marine riser in cross-flow, subject to vortex-induced vibrations (VIV). VIV are driven by an absolute instability in the flow, which imposes a freq… ▽ More We develop a fast multi-fidelity modeling method for very complex correlations between high- and low-fidelity data by working in modal space to extract the proper correlation function. We apply this method to infer the amplitude of motion of a flexible marine riser in cross-flow, subject to vortex-induced vibrations (VIV). VIV are driven by an absolute instability in the flow, which imposes a frequency (Strouhal) law that requires a matching with the impedance of the structure; this matching is easily achieved because of the rapid parametric variation of the added mass force. As a result, the wavenumber of the riser spatial response is within narrow bands of uncertainty. Hence, an error in wavenumber prediction can cause significant phase-related errors in the shape of the amplitude of response along the riser, rendering correlation between low- and high-fidelity data very complex. Working in modal space as outlined herein, dense data from low-fidelity data, provided by the semi-empirical computer code VIVA, can correlate in modal space with few high-fidelity data, obtained from experiments or fully-resolved CFD simulations, to correct both phase and amplitude and provide predictions that agree very well overall with the correct shape of the amplitude response. We also quantify the uncertainty in the prediction using Bayesian modeling and exploit this uncertainty to formulate an active learning strategy for the best possible location of the sensors providing the high fidelity measurements. △ Less

Submitted 24 December, 2020; originally announced December 2020.

Comments: 26 pages, 13 figures

Multi-fidelity Bayesian Neural Networks: Algorithms and Applications

Authors: Xuhui Meng, Hessam Babaee, George Em Karniadakis

Abstract: We propose a new class of Bayesian neural networks (BNNs) that can be trained using noisy data of variable fidelity, and we apply them to learn function approximations as well as to solve inverse problems based on partial differential equations (PDEs). These multi-fidelity BNNs consist of three neural networks: The first is a fully connected neural network, which is trained following the maximum a… ▽ More We propose a new class of Bayesian neural networks (BNNs) that can be trained using noisy data of variable fidelity, and we apply them to learn function approximations as well as to solve inverse problems based on partial differential equations (PDEs). These multi-fidelity BNNs consist of three neural networks: The first is a fully connected neural network, which is trained following the maximum a posteriori probability (MAP) method to fit the low-fidelity data; the second is a Bayesian neural network employed to capture the cross-correlation with uncertainty quantification between the low- and high-fidelity data; and the last one is the physics-informed neural network, which encodes the physical laws described by PDEs. For the training of the last two neural networks, we use the Hamiltonian Monte Carlo method to estimate accurately the posterior distributions for the corresponding hyperparameters. We demonstrate the accuracy of the present method using synthetic data as well as real measurements. Specifically, we first approximate a one- and four-dimensional function, and then infer the reaction rates in one- and two-dimensional diffusion-reaction systems. Moreover, we infer the sea surface temperature (SST) in the Massachusetts and Cape Cod Bays using satellite images and in-situ measurements. Taken together, our results demonstrate that the present method can capture both linear and nonlinear correlation between the low- and high-fideilty data adaptively, identify unknown parameters in PDEs, and quantify uncertainties in predictions, given a few scattered noisy high-fidelity data. Finally, we demonstrate that we can effectively and efficiently reduce the uncertainties and hence enhance the prediction accuracy with an active learning approach, using as examples a specific one-dimensional function approximation and an inverse PDE problem. △ Less

Submitted 18 December, 2020; originally announced December 2020.

Comments: 31 pages, 11 figures

Operator learning for predicting multiscale bubble growth dynamics

Authors: Chensen Lin, Zhen Li, Lu Lu, Shengze Cai, Martin Maxey, George Em Karniadakis

Abstract: Simulating and predicting multiscale problems that couple multiple physics and dynamics across many orders of spatiotemporal scales is a great challenge that has not been investigated systematically by deep neural networks (DNNs). Herein, we develop a framework based on operator regression, the so-called deep operator network (DeepONet), with the long term objective to simplify multiscale modeling… ▽ More Simulating and predicting multiscale problems that couple multiple physics and dynamics across many orders of spatiotemporal scales is a great challenge that has not been investigated systematically by deep neural networks (DNNs). Herein, we develop a framework based on operator regression, the so-called deep operator network (DeepONet), with the long term objective to simplify multiscale modeling by avoiding the fragile and time-consuming "hand-shaking" interface algorithms for stitching together heterogeneous descriptions of multiscale phenomena. To this end, as a first step, we investigate if a DeepONet can learn the dynamics of different scale regimes, one at the deterministic macroscale and the other at the stochastic microscale regime with inherent thermal fluctuations. Specifically, we test the effectiveness and accuracy of DeepONet in predicting multirate bubble growth dynamics, which is described by a Rayleigh-Plesset (R-P) equation at the macroscale and modeled as a stochastic nucleation and cavitation process at the microscale by dissipative particle dynamics (DPD). Taken together, our findings demonstrate that DeepONets can be employed to unify the macroscale and microscale models of the multirate bubble growth problem, hence providing new insight into the role of operator regression via DNNs in tackling realistic multiscale problems and in simplifying modeling with heterogeneous descriptions. △ Less

Submitted 23 December, 2020; originally announced December 2020.

DeepM&Mnet for hypersonics: Predicting the coupled flow and finite-rate chemistry behind a normal shock using neural-network approximation of operators

Authors: Zhiping Mao, Lu Lu, Olaf Marxen, Tamer A. Zaki, George E. Karniadakis

Abstract: In high-speed flow past a normal shock, the fluid temperature rises rapidly triggering downstream chemical dissociation reactions. The chemical changes lead to appreciable changes in fluid properties, and these coupled multiphysics and the resulting multiscale dynamics are challenging to resolve numerically. Using conventional computational fluid dynamics (CFD) requires excessive computing cost. H… ▽ More In high-speed flow past a normal shock, the fluid temperature rises rapidly triggering downstream chemical dissociation reactions. The chemical changes lead to appreciable changes in fluid properties, and these coupled multiphysics and the resulting multiscale dynamics are challenging to resolve numerically. Using conventional computational fluid dynamics (CFD) requires excessive computing cost. Here, we propose a totally new efficient approach, assuming that some sparse measurements of the state variables are available that can be seamlessly integrated in the simulation algorithm. We employ a special neural network for approximating nonlinear operators, the DeepONet, which is used to predict separately each individual field, given inputs from the rest of the fields of the coupled multiphysics system. We demonstrate the effectiveness of DeepONet by predicting five species in the non-equilibrium chemistry downstream of a normal shock at high Mach numbers as well as the velocity and temperature fields. We show that upon training, DeepONets can be over five orders of magnitude faster than the CFD solver employed to generate the training data and yield good accuracy for unseen Mach numbers within the range of training. Outside this range, DeepONet can still predict accurately and fast if a few sparse measurements are available. We then propose a composite supervised neural network, DeepM&Mnet, that uses multiple pre-trained DeepONets as building blocks and scattered measurements to infer the set of all seven fields in the entire domain of interest. Two DeepM&Mnet architectures are tested, and we demonstrate the accuracy and capacity for efficient data assimilation. DeepM&Mnet is simple and general: it can be employed to construct complex multiphysics and multiscale models and assimilate sparse measurements using pre-trained DeepONets in a "plug-and-play" mode. △ Less

Submitted 1 November, 2020; originally announced November 2020.

Comments: 30 pages, 17 figures

Physics-informed neural networks for solving forward and inverse flow problems via the Boltzmann-BGK formulation

Authors: Qin Lou, Xuhui Meng, George Em Karniadakis

Abstract: In this study, we employ physics-informed neural networks (PINNs) to solve forward and inverse problems via the Boltzmann-BGK formulation (PINN-BGK), enabling PINNs to model flows in both the continuum and rarefied regimes. In particular, the PINN-BGK is composed of three sub-networks, i.e., the first for approximating the equilibrium distribution function, the second for approximating the non-equ… ▽ More In this study, we employ physics-informed neural networks (PINNs) to solve forward and inverse problems via the Boltzmann-BGK formulation (PINN-BGK), enabling PINNs to model flows in both the continuum and rarefied regimes. In particular, the PINN-BGK is composed of three sub-networks, i.e., the first for approximating the equilibrium distribution function, the second for approximating the non-equilibrium distribution function, and the third one for encoding the Boltzmann-BGK equation as well as the corresponding boundary/initial conditions. By minimizing the residuals of the governing equations and the mismatch between the predicted and provided boundary/initial conditions, we can approximate the Boltzmann-BGK equation for both continuous and rarefied flows. For forward problems, the PINN-BGK is utilized to solve various benchmark flows given boundary/initial conditions, e.g., Kovasznay flow, Taylor-Green flow, cavity flow, and micro Couette flow for Knudsen number up to 5. For inverse problems, we focus on rarefied flows in which accurate boundary conditions are difficult to obtain. We employ the PINN-BGK to infer the flow field in the entire computational domain given a limited number of interior scattered measurements on the velocity with unknown boundary conditions. Results for the two-dimensional micro Couette and micro cavity flows with Knudsen numbers ranging from 0.1 to 10 indicate that the PINN-BGK can infer the velocity field in the entire domain with good accuracy. Finally, we also present some results on using transfer learning to accelerate the training process. Specifically, we can obtain a three-fold speedup compared to the standard training process (e.g., Adam plus L-BFGS-B) for the two-dimensional flow problems considered in our work. △ Less

Submitted 18 October, 2020; originally announced October 2020.

Comments: 30 pages, 11 figures

DeepM&Mnet: Inferring the electroconvection multiphysics fields based on operator approximation by neural networks

Authors: Shengze Cai, Zhicheng Wang, Lu Lu, Tamer A Zaki, George Em Karniadakis

Abstract: Electroconvection is a multiphysics problem involving coupling of the flow field with the electric field as well as the cation and anion concentration fields. For small Debye lengths, very steep boundary layers are developed, but standard numerical methods can simulate the different regimes quite accurately. Here, we use electroconvection as a benchmark problem to put forward a new data assimilati… ▽ More Electroconvection is a multiphysics problem involving coupling of the flow field with the electric field as well as the cation and anion concentration fields. For small Debye lengths, very steep boundary layers are developed, but standard numerical methods can simulate the different regimes quite accurately. Here, we use electroconvection as a benchmark problem to put forward a new data assimilation framework, the DeepM&Mnet, for simulating multiphysics and multiscale problems at speeds much faster than standard numerical methods using pre-trained neural networks (NNs). We first pre-train DeepONets that can predict independently each field, given general inputs from the rest of the fields of the coupled system. DeepONets can approximate nonlinear operators and are composed of two sub-networks, a branch net for the input fields and a trunk net for the locations of the output field. DeepONets, which are extremely fast, are used as building blocks in the DeepM&Mnet and form constraints for the multiphysics solution along with some sparse available measurements of any of the fields. We demonstrate the new methodology and document the accuracy of each individual DeepONet, and subsequently we present two different DeepM&Mnet architectures that infer accurately and efficiently 2D electroconvection fields for unseen electric potentials. The DeepM&Mnet framework is general and can be applied for building any complex multiphysics and multiscale models based on very few measurements using pre-trained DeepONets in a plug-and-play mode. △ Less

Submitted 27 September, 2020; originally announced September 2020.

Solving Inverse Stochastic Problems from Discrete Particle Observations Using the Fokker-Planck Equation and Physics-informed Neural Networks

Authors: Xiaoli Chen, Liu Yang, Jinqiao Duan, George Em Karniadakis

Abstract: The Fokker-Planck (FP) equation governing the evolution of the probability density function (PDF) is applicable to many disciplines but it requires specification of the coefficients for each case, which can be functions of space-time and not just constants, hence requiring the development of a data-driven modeling approach. When the data available is directly on the PDF, then there exist methods f… ▽ More The Fokker-Planck (FP) equation governing the evolution of the probability density function (PDF) is applicable to many disciplines but it requires specification of the coefficients for each case, which can be functions of space-time and not just constants, hence requiring the development of a data-driven modeling approach. When the data available is directly on the PDF, then there exist methods for inverse problems that can be employed to infer the coefficients and thus determine the FP equation and subsequently obtain its solution. Herein, we address a more realistic scenario, where only sparse data are given on the particles' positions at a few time instants, which are not sufficient to accurately construct directly the PDF even at those times from existing methods, e.g., kernel estimation algorithms. To this end, we develop a general framework based on physics-informed neural networks (PINNs) that introduces a new loss function using the Kullback-Leibler divergence to connect the stochastic samples with the FP equation, to simultaneously learn the equation and infer the multi-dimensional PDF at all times. In particular, we consider two types of inverse problems, type I where the FP equation is known but the initial PDF is unknown, and type II in which, in addition to unknown initial PDF, the drift and diffusion terms are also unknown. In both cases, we investigate problems with either Brownian or Levy noise or a combination of both. We demonstrate the new PINN framework in detail in the one-dimensional case (1D) but we also provide results for up to 5D demonstrating that we can infer both the FP equation and} dynamics simultaneously at all times with high accuracy using only very few discrete observations of the particles. △ Less

Submitted 24 August, 2020; originally announced August 2020.

Comments: The first two authors contributed equally to this paper. Corresponding author: George Em Karniadakis

Non-invasive Inference of Thrombus Material Properties with Physics-informed Neural Networks

Authors: Minglang Yin, Xiaoning Zheng, Jay D. Humphrey, George Em Karniadakis

Abstract: We employ physics-informed neural networks (PINNs) to infer properties of biological materials using synthetic data. In particular, we successfully apply PINNs on inferring the thrombus permeability and visco-elastic modulus from thrombus deformation data, which can be described by the fourth-order Cahn-Hilliard and Navier-Stokes Equations. In PINNs, the partial differential equations are encoded… ▽ More We employ physics-informed neural networks (PINNs) to infer properties of biological materials using synthetic data. In particular, we successfully apply PINNs on inferring the thrombus permeability and visco-elastic modulus from thrombus deformation data, which can be described by the fourth-order Cahn-Hilliard and Navier-Stokes Equations. In PINNs, the partial differential equations are encoded into the loss function, where partial derivatives can be obtained through automatic differentiation (AD). In addition, to tackling the challenge of calculating the fourth-order derivative in the Cahn-Hilliard equation with AD, we introduce an auxiliary network along with the main neural network to approximate the second-derivative of the energy potential term. Our model can predict simultaneously unknown parameters and velocity, pressure, and deformation gradient fields by merely training with partial information among all data, i.e., phase-field and pressure measurements, and is also highly flexible in sampling within the spatio-temporal domain for data acquisition. We validate our model by numerical solutions from the spectral/\textit{hp} element method (SEM) and demonstrate its robustness by training it with noisy measurements. Our results show that PINNs can accurately infer the material properties with noisy synthetic data, and thus they have great potential for inferring these properties from experimental multi-modality and multi-fidelity data. △ Less

Submitted 22 May, 2020; originally announced May 2020.

Active- and transfer-learning applied to microscale-macroscale coupling to simulate viscoelastic flows

Authors: Lifei Zhao, Zhen Li, Zhicheng Wang, Bruce Caswell, Jie Ouyang, George Em Karniadakis

Abstract: Active- and transfer-learning are applied to polymer flows for the multiscale discovery of effective constitutive approximations required in viscoelastic flow simulation. The result is macroscopic rheology directly connected to a microstructural model. Micro and macroscale simulations are adaptively coupled by means of Gaussian process regression to run the expensive microscale computations only a… ▽ More Active- and transfer-learning are applied to polymer flows for the multiscale discovery of effective constitutive approximations required in viscoelastic flow simulation. The result is macroscopic rheology directly connected to a microstructural model. Micro and macroscale simulations are adaptively coupled by means of Gaussian process regression to run the expensive microscale computations only as necessary. This active-learning guided multiscale method can automatically detect the inaccuracy of the learned constitutive closure and initiate simulations at new sampling points informed by proper acquisition functions, leading to an autonomic microscale-macroscale coupled system. Also, we develop a new dissipative particle dynamics model with the range of interaction cutoff between particles allowed to vary with the local strain-rate invariant, which is able to capture both the shear-thinning viscosity and the normal stress difference functions consistent with rheological experiments for aqueous polyacrylamide solutions. Our numerical experiments demonstrate the effectiveness of using active- and transfer-learning schemes to on-the-fly couple a spectral element solver and a mesoscopic particle-based simulator, and verify that the microscale-macroscale coupled model with effective constitutive closure learned from microscopic dynamics can outperform empirical constitutive models compared to experimental observations. The effective closure learned in a channel simulation is then transferred directly to the flow past a circular cylinder, where the results show that only two additional microscopic simulations are required to achieve a satisfactory constitutive model to once again close the continuum equations. This new paradigm of active- and transfer-learning for multiscale modeling is readily applicable to other microscale-macroscale coupled simulations of complex fluids and other materials. △ Less

Submitted 9 May, 2020; originally announced May 2020.

Comments: 26 pages, 16 figures

nPINNs: nonlocal Physics-Informed Neural Networks for a parametrized nonlocal universal Laplacian operator. Algorithms and Applications

Authors: Guofei Pang, Marta D'Elia, Michael Parks, George E. Karniadakis

Abstract: Physics-informed neural networks (PINNs) are effective in solving inverse problems based on differential and integral equations with sparse, noisy, unstructured, and multi-fidelity data. PINNs incorporate all available information into a loss function, thus recasting the original problem into an optimization problem. In this paper, we extend PINNs to parameter and function inference for integral e… ▽ More Physics-informed neural networks (PINNs) are effective in solving inverse problems based on differential and integral equations with sparse, noisy, unstructured, and multi-fidelity data. PINNs incorporate all available information into a loss function, thus recasting the original problem into an optimization problem. In this paper, we extend PINNs to parameter and function inference for integral equations such as nonlocal Poisson and nonlocal turbulence models, and we refer to them as nonlocal PINNs (nPINNs). The contribution of the paper is three-fold. First, we propose a unified nonlocal operator, which converges to the classical Laplacian as one of the operator parameters, the nonlocal interaction radius $δでるた$ goes to zero, and to the fractional Laplacian as $δでるた$ goes to infinity. This universal operator forms a super-set of classical Laplacian and fractional Laplacian operators and, thus, has the potential to fit a broad spectrum of data sets. We provide theoretical convergence rates with respect to $δでるた$ and verify them via numerical experiments. Second, we use nPINNs to estimate the two parameters, $δでるた$ and $αあるふぁ$. The strong non-convexity of the loss function yielding multiple (good) local minima reveals the occurrence of the operator mimicking phenomenon: different pairs of estimated parameters could produce multiple solutions of comparable accuracy. Third, we propose another nonlocal operator with spatially variable order $αあるふぁ(y)$, which is more suitable for modeling turbulent Couette flow. Our results show that nPINNs can jointly infer this function as well as $δでるた$. Also, these parameters exhibit a universal behavior with respect to the Reynolds number, a finding that contributes to our understanding of nonlocal interactions in wall-bounded turbulence. △ Less

Submitted 8 April, 2020; originally announced April 2020.

Comments: 31 pages, 20 figures

Report number: SAND2020-3980 MSC Class: 34B10; 45P05; 92B20; 35Q93; 76F10

NSFnets (Navier-Stokes Flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations

Authors: Xiaowei Jin, Shengze Cai, Hui Li, George Em Karniadakis

Abstract: We employ physics-informed neural networks (PINNs) to simulate the incompressible flows ranging from laminar to turbulent flows. We perform PINN simulations by considering two different formulations of the Navier-Stokes equations: the velocity-pressure (VP) formulation and the vorticity-velocity (VV) formulation. We refer to these specific PINNs for the Navier-Stokes flow nets as NSFnets. Analytic… ▽ More We employ physics-informed neural networks (PINNs) to simulate the incompressible flows ranging from laminar to turbulent flows. We perform PINN simulations by considering two different formulations of the Navier-Stokes equations: the velocity-pressure (VP) formulation and the vorticity-velocity (VV) formulation. We refer to these specific PINNs for the Navier-Stokes flow nets as NSFnets. Analytical solutions and direct numerical simulation (DNS) databases provide proper initial and boundary conditions for the NSFnet simulations. The spatial and temporal coordinates are the inputs of the NSFnets, while the instantaneous velocity and pressure fields are the outputs for the VP-NSFnet, and the instantaneous velocity and vorticity fields are the outputs for the VV-NSFnet. These two different forms of the Navier-Stokes equations together with the initial and boundary conditions are embedded into the loss function of the PINNs. No data is provided for the pressure to the VP-NSFnet, which is a hidden state and is obtained via the incompressibility constraint without splitting the equations. We obtain good accuracy of the NSFnet simulation results upon convergence of the loss function, verifying that NSFnets can effectively simulate complex incompressible flows using either the VP or the VV formulations. We also perform a systematic study on the weights used in the loss function for the data/physics components and investigate a new way of computing the weights dynamically to accelerate training and enhance accuracy. Our results suggest that the accuracy of NSFnets, for both laminar and turbulent flows, can be improved with proper tuning of weights (manual or dynamic) in the loss function. △ Less

Submitted 13 March, 2020; originally announced March 2020.

Reinforcement Learning for Active Flow Control in Experiments

Authors: Dixia Fan, Liu Yang, Michael S Triantafyllou, George Em Karniadakis

Abstract: We demonstrate experimentally the feasibility of applying reinforcement learning (RL) in flow control problems by automatically discovering active control strategies without any prior knowledge of the flow physics. We consider the turbulent flow past a circular cylinder with the aim of reducing the cylinder drag force or maximizing the power gain efficiency by properly selecting the rotational spe… ▽ More We demonstrate experimentally the feasibility of applying reinforcement learning (RL) in flow control problems by automatically discovering active control strategies without any prior knowledge of the flow physics. We consider the turbulent flow past a circular cylinder with the aim of reducing the cylinder drag force or maximizing the power gain efficiency by properly selecting the rotational speed of two small diameter cylinders, parallel to and located downstream of the larger cylinder. Given properly designed rewards and noise reduction techniques, after tens of towing experiments, the RL agent could discover the optimal control strategy, comparable to the optimal static control. While RL has been found to be effective in recent computer flow simulation studies, this is the first time that its effectiveness is demonstrated experimentally, paving the way for exploring new optimal active flow control strategies in complex fluid mechanics applications. △ Less

Submitted 6 March, 2020; originally announced March 2020.

Comments: The first two authors contributed equally to this work

SympNets: Intrinsic structure-preserving symplectic networks for identifying Hamiltonian systems

Authors: Pengzhan Jin, Zhen Zhang, Aiqing Zhu, Yifa Tang, George Em Karniadakis

Abstract: We propose new symplectic networks (SympNets) for identifying Hamiltonian systems from data based on a composition of linear, activation and gradient modules. In particular, we define two classes of SympNets: the LA-SympNets composed of linear and activation modules, and the G-SympNets composed of gradient modules. Correspondingly, we prove two new universal approximation theorems that demonstrate… ▽ More We propose new symplectic networks (SympNets) for identifying Hamiltonian systems from data based on a composition of linear, activation and gradient modules. In particular, we define two classes of SympNets: the LA-SympNets composed of linear and activation modules, and the G-SympNets composed of gradient modules. Correspondingly, we prove two new universal approximation theorems that demonstrate that SympNets can approximate arbitrary symplectic maps based on appropriate activation functions. We then perform several experiments including the pendulum, double pendulum and three-body problems to investigate the expressivity and the generalization ability of SympNets. The simulation results show that even very small size SympNets can generalize well, and are able to handle both separable and non-separable Hamiltonian systems with data points resulting from short or long time steps. In all the test cases, SympNets outperform the baseline models, and are much faster in training and prediction. We also develop an extended version of SympNets to learn the dynamics from irregularly sampled data. This extended version of SympNets can be thought of as a universal model representing the solution to an arbitrary Hamiltonian system. △ Less

Submitted 19 August, 2020; v1 submitted 11 January, 2020; originally announced January 2020.