Showing 1–47 of 47 results for author: Jafarpour, S

$\texttt{immrax}$: A Parallelizable and Differentiable Toolbox for Interval Analysis and Mixed Monotone Reachability in JAX

Authors: Akash Harapanahalli, Saber Jafarpour, Samuel Coogan

Abstract: We present an implementation of interval analysis and mixed monotone interval reachability analysis as function transforms in Python, fully composable with the computational framework JAX. The resulting toolbox inherits several key features from JAX, including computational efficiency through Just-In-Time Compilation, GPU acceleration for quick parallelized computations, and Automatic Differentiab… ▽ More We present an implementation of interval analysis and mixed monotone interval reachability analysis as function transforms in Python, fully composable with the computational framework JAX. The resulting toolbox inherits several key features from JAX, including computational efficiency through Just-In-Time Compilation, GPU acceleration for quick parallelized computations, and Automatic Differentiability. We demonstrate the toolbox's performance on several case studies, including a reachability problem on a vehicle model controlled by a neural network, and a robust closed-loop optimal control problem for a swinging pendulum. △ Less

Submitted 30 April, 2024; v1 submitted 21 January, 2024; originally announced January 2024.

A Contracting Dynamical System Perspective toward Interval Markov Decision Processes

Authors: Saber Jafarpour, Samuel Coogan

Abstract: Interval Markov decision processes are a class of Markov models where the transition probabilities between the states belong to intervals. In this paper, we study the problem of efficient estimation of the optimal policies in Interval Markov Decision Processes (IMDPs) with continuous action-space. Given an IMDP, we show that the pessimistic (resp. the optimistic) value iterations, i.e., the value… ▽ More Interval Markov decision processes are a class of Markov models where the transition probabilities between the states belong to intervals. In this paper, we study the problem of efficient estimation of the optimal policies in Interval Markov Decision Processes (IMDPs) with continuous action-space. Given an IMDP, we show that the pessimistic (resp. the optimistic) value iterations, i.e., the value iterations under the assumption of a competitive adversary (resp. cooperative agent), are monotone dynamical systems and are contracting with respect to the $\ell_{\infty}$-norm. Inspired by this dynamical system viewpoint, we introduce another IMDP, called the action-space relaxation IMDP. We show that the action-space relaxation IMDP has two key features: (i) its optimal value is an upper bound for the optimal value of the original IMDP, and (ii) its value iterations can be efficiently solved using tools and techniques from convex optimization. We then consider the policy optimization problems at each step of the value iterations as a feedback controller of the value function. Using this system-theoretic perspective, we propose an iteration-distributed implementation of the value iterations for approximating the optimal value of the action-space relaxation IMDP. △ Less

Submitted 16 September, 2023; originally announced September 2023.

Forward Invariance in Neural Network Controlled Systems

Authors: Akash Harapanahalli, Saber Jafarpour, Samuel Coogan

Abstract: We present a framework based on interval analysis and monotone systems theory to certify and search for forward invariant sets in nonlinear systems with neural network controllers. The framework (i) constructs localized first-order inclusion functions for the closed-loop system using Jacobian bounds and existing neural network verification tools; (ii) builds a dynamical embedding system where its… ▽ More We present a framework based on interval analysis and monotone systems theory to certify and search for forward invariant sets in nonlinear systems with neural network controllers. The framework (i) constructs localized first-order inclusion functions for the closed-loop system using Jacobian bounds and existing neural network verification tools; (ii) builds a dynamical embedding system where its evaluation along a single trajectory directly corresponds with a nested family of hyper-rectangles provably converging to an attractive set of the original system; (iii) utilizes linear transformations to build families of nested paralleletopes with the same properties. The framework is automated in Python using our interval analysis toolbox $\texttt{npinterval}$, in conjunction with the symbolic arithmetic toolbox $\texttt{sympy}$, demonstrated on an $8$-dimensional leader-follower system. △ Less

Submitted 9 December, 2023; v1 submitted 16 September, 2023; originally announced September 2023.

Efficient Interaction-Aware Interval Analysis of Neural Network Feedback Loops

Authors: Saber Jafarpour, Akash Harapanahalli, Samuel Coogan

Abstract: In this paper, we propose a computationally efficient framework for interval reachability of systems with neural network controllers. Our approach leverages inclusion functions for the open-loop system and the neural network controller to embed the closed-loop system into a larger-dimensional embedding system, where a single trajectory over-approximates the original system's behavior under uncerta… ▽ More In this paper, we propose a computationally efficient framework for interval reachability of systems with neural network controllers. Our approach leverages inclusion functions for the open-loop system and the neural network controller to embed the closed-loop system into a larger-dimensional embedding system, where a single trajectory over-approximates the original system's behavior under uncertainty. We propose two methods for constructing closed-loop embedding systems, which account for the interactions between the system and the controller in different ways. The interconnection-based approach considers the worst-case evolution of each coordinate separately by substituting the neural network inclusion function into the open-loop inclusion function. The interaction-based approach uses novel Jacobian-based inclusion functions to capture the first-order interactions between the open-loop system and the controller by leveraging state-of-the-art neural network verifiers. Finally, we implement our approach in a Python framework called ReachMM to demonstrate its efficiency and scalability on benchmarks and examples ranging to $200$ state dimensions. △ Less

Submitted 27 June, 2024; v1 submitted 27 July, 2023; originally announced July 2023.

A Toolbox for Fast Interval Arithmetic in numpy with an Application to Formal Verification of Neural Network Controlled Systems

Authors: Akash Harapanahalli, Saber Jafarpour, Samuel Coogan

Abstract: In this paper, we present a toolbox for interval analysis in numpy, with an application to formal verification of neural network controlled systems. Using the notion of natural inclusion functions, we systematically construct interval bounds for a general class of mappings. The toolbox offers efficient computation of natural inclusion functions using compiled C code, as well as a familiar interfac… ▽ More In this paper, we present a toolbox for interval analysis in numpy, with an application to formal verification of neural network controlled systems. Using the notion of natural inclusion functions, we systematically construct interval bounds for a general class of mappings. The toolbox offers efficient computation of natural inclusion functions using compiled C code, as well as a familiar interface in numpy with its canonical features, such as n-dimensional arrays, matrix/vector operations, and vectorization. We then use this toolbox in formal verification of dynamical systems with neural network controllers, through the composition of their inclusion functions. △ Less

Submitted 27 June, 2023; originally announced June 2023.

Contraction-Guided Adaptive Partitioning for Reachability Analysis of Neural Network Controlled Systems

Authors: Akash Harapanahalli, Saber Jafarpour, Samuel Coogan

Abstract: In this paper, we present a contraction-guided adaptive partitioning algorithm for improving interval-valued robust reachable set estimates in a nonlinear feedback loop with a neural network controller and disturbances. Based on an estimate of the contraction rate of over-approximated intervals, the algorithm chooses when and where to partition. Then, by leveraging a decoupling of the neural netwo… ▽ More In this paper, we present a contraction-guided adaptive partitioning algorithm for improving interval-valued robust reachable set estimates in a nonlinear feedback loop with a neural network controller and disturbances. Based on an estimate of the contraction rate of over-approximated intervals, the algorithm chooses when and where to partition. Then, by leveraging a decoupling of the neural network verification step and reachability partitioning layers, the algorithm can provide accuracy improvements for little computational cost. This approach is applicable with any sufficiently accurate open-loop interval-valued reachability estimation technique and any method for bounding the input-output behavior of a neural network. Using contraction-based robustness analysis, we provide guarantees of the algorithm's performance with mixed monotone reachability. Finally, we demonstrate the algorithm's performance through several numerical simulations and compare it with existing methods in the literature. In particular, we report a sizable improvement in the accuracy of reachable set estimation in a fraction of the runtime as compared to state-of-the-art methods. △ Less

Submitted 9 December, 2023; v1 submitted 7 April, 2023; originally announced April 2023.

Non-Euclidean Monotone Operator Theory and Applications

Authors: Alexander Davydov, Saber Jafarpour, Anton V. Proskurnikov, Francesco Bullo

Abstract: While monotone operator theory is traditionally studied on Hilbert spaces, many interesting problems in data science and machine learning arise naturally in finite-dimensional vector spaces endowed with non-Euclidean norms, such as diagonally-weighted $\ell_1$ or $\ell_\infty$ norms. This paper provides a natural generalization of monotone operator theory to finite-dimensional non-Euclidean spaces… ▽ More While monotone operator theory is traditionally studied on Hilbert spaces, many interesting problems in data science and machine learning arise naturally in finite-dimensional vector spaces endowed with non-Euclidean norms, such as diagonally-weighted $\ell_1$ or $\ell_\infty$ norms. This paper provides a natural generalization of monotone operator theory to finite-dimensional non-Euclidean spaces. The key tools are weak pairings and logarithmic norms. We show that the resolvent and reflected resolvent operators of non-Euclidean monotone mappings exhibit similar properties to their counterparts in Hilbert spaces. Furthermore, classical iterative methods and splitting methods for finding zeros of monotone operators are shown to converge in the non-Euclidean case. We apply our theory to equilibrium computation and Lipschitz constant estimation of recurrent neural networks, obtaining novel iterations and tighter upper bounds via forward-backward splitting. △ Less

Submitted 20 March, 2023; originally announced March 2023.

Comments: arXiv admin note: text overlap with arXiv:2204.01877

Interval Reachability of Nonlinear Dynamical Systems with Neural Network Controllers

Authors: Saber Jafarpour, Akash Harapanahalli, Samuel Coogan

Abstract: This paper proposes a computationally efficient framework, based on interval analysis, for rigorous verification of nonlinear continuous-time dynamical systems with neural network controllers. Given a neural network, we use an existing verification algorithm to construct inclusion functions for its input-output behavior. Inspired by mixed monotone theory, we embed the closed-loop dynamics into a l… ▽ More This paper proposes a computationally efficient framework, based on interval analysis, for rigorous verification of nonlinear continuous-time dynamical systems with neural network controllers. Given a neural network, we use an existing verification algorithm to construct inclusion functions for its input-output behavior. Inspired by mixed monotone theory, we embed the closed-loop dynamics into a larger system using an inclusion function of the neural network and a decomposition function of the open-loop system. This embedding provides a scalable approach for safety analysis of the neural control loop while preserving the nonlinear structure of the system. We show that one can efficiently compute hyper-rectangular over-approximations of the reachable sets using a single trajectory of the embedding system. We design an algorithm to leverage this computational advantage through partitioning strategies, improving our reachable set estimates while balancing its runtime with tunable parameters. We demonstrate the performance of this algorithm through two case studies. First, we demonstrate this method's strength in complex nonlinear environments. Then, we show that our approach matches the performance of the state-of-the art verification algorithm for linear discretized systems. △ Less

Submitted 7 August, 2023; v1 submitted 19 January, 2023; originally announced January 2023.

Comments: Extended L4DC version with proofs

Monotonicity and Contraction on Polyhedral Cones

Authors: Saber Jafarpour, Samuel Coogan

Abstract: In this note, we study monotone dynamical systems with respect to polyhedral cones. Using the half-space representation and the vertex representation, we propose three equivalent conditions to certify monotonicity of a dynamical system with respect to a polyhedral cone. We then introduce the notion of gauge norm associated with a cone and provide closed-from formulas for computing gauge norms asso… ▽ More In this note, we study monotone dynamical systems with respect to polyhedral cones. Using the half-space representation and the vertex representation, we propose three equivalent conditions to certify monotonicity of a dynamical system with respect to a polyhedral cone. We then introduce the notion of gauge norm associated with a cone and provide closed-from formulas for computing gauge norms associated with polyhedral cones. A key feature of gauge norms is that contractivity of monotone systems with respect to them can be efficiently characterized using simple inequalities. This result generalizes the well-known criteria for Hurwitzness of Metzler matrices and provides a scalable approach to search for Lyapunov functions of monotone systems with respect to polyhedral cones. Finally, we study the applications of our results in transient stability of dynamic flow networks and in scalable control design with safety guarantees. △ Less

Submitted 7 August, 2023; v1 submitted 20 October, 2022; originally announced October 2022.

Robust Training and Verification of Implicit Neural Networks: A Non-Euclidean Contractive Approach

Authors: Saber Jafarpour, Alexander Davydov, Matthew Abate, Francesco Bullo, Samuel Coogan

Abstract: This paper proposes a theoretical and computational framework for training and robustness verification of implicit neural networks based upon non-Euclidean contraction theory. The basic idea is to cast the robustness analysis of a neural network as a reachability problem and use (i) the $\ell_{\infty}$-norm input-output Lipschitz constant and (ii) the tight inclusion function of the network to ove… ▽ More This paper proposes a theoretical and computational framework for training and robustness verification of implicit neural networks based upon non-Euclidean contraction theory. The basic idea is to cast the robustness analysis of a neural network as a reachability problem and use (i) the $\ell_{\infty}$-norm input-output Lipschitz constant and (ii) the tight inclusion function of the network to over-approximate its reachable sets. First, for a given implicit neural network, we use $\ell_{\infty}$-matrix measures to propose sufficient conditions for its well-posedness, design an iterative algorithm to compute its fixed points, and provide upper bounds for its $\ell_\infty$-norm input-output Lipschitz constant. Second, we introduce a related embedded network and show that the embedded network can be used to provide an $\ell_\infty$-norm box over-approximation of the reachable sets of the original network. Moreover, we use the embedded network to design an iterative algorithm for computing the upper bounds of the original system's tight inclusion function. Third, we use the upper bounds of the Lipschitz constants and the upper bounds of the tight inclusion functions to design two algorithms for the training and robustness verification of implicit neural networks. Finally, we apply our algorithms to train implicit neural networks on the MNIST dataset and compare the robustness of our models with the models trained via existing approaches in the literature. △ Less

Submitted 7 August, 2022; originally announced August 2022.

Comments: arXiv admin note: text overlap with arXiv:2112.05310

Network Critical Slowing Down: Data-Driven Detection of Critical Transitions in Nonlinear Networks

Authors: Mohammad Pirani, Saber Jafarpour

Abstract: In a Nature article, Scheffer et al. presented a novel data-driven framework to predict critical transitions in complex systems. These transitions, which may stem from failures, degradation, or adversarial actions, have been attributed to bifurcations in the nonlinear dynamics. Their approach was built upon the phenomenon of critical slowing down, i.e., slow recovery in response to small perturbat… ▽ More In a Nature article, Scheffer et al. presented a novel data-driven framework to predict critical transitions in complex systems. These transitions, which may stem from failures, degradation, or adversarial actions, have been attributed to bifurcations in the nonlinear dynamics. Their approach was built upon the phenomenon of critical slowing down, i.e., slow recovery in response to small perturbations near bifurcations. We extend their approach to detect and localize critical transitions in nonlinear networks. By introducing the notion of network critical slowing down, the objective of this paper is to detect that the network is undergoing a bifurcation only by analyzing its signatures from measurement data. We focus on two classes of widely-used nonlinear networks: (1) Kuramoto model for the synchronization of coupled oscillators and (2) attraction-repulsion dynamics in swarms, each of which presents a specific type of bifurcation. Based on the phenomenon of critical slowing down, we study the asymptotic behavior of the perturbed system away and close to the bifurcation and leverage this fact to develop a deterministic method to detect and identify critical transitions in nonlinear networks. Furthermore, we study the state covariance matrix subject to a stochastic noise process away and close to the bifurcation and use it to develop a stochastic framework for detecting critical transitions. Our simulation results show the strengths and limitations of the methods. △ Less

Submitted 7 August, 2022; originally announced August 2022.

Non-Euclidean Monotone Operator Theory with Applications to Recurrent Neural Networks

Authors: Alexander Davydov, Saber Jafarpour, Anton V. Proskurnikov, Francesco Bullo

Abstract: We provide a novel transcription of monotone operator theory to the non-Euclidean finite-dimensional spaces $\ell_1$ and $\ell_{\infty}$. We first establish properties of mappings which are monotone with respect to the non-Euclidean norms $\ell_1$ or $\ell_{\infty}$. In analogy with their Euclidean counterparts, mappings which are monotone with respect to a non-Euclidean norm are amenable to numer… ▽ More We provide a novel transcription of monotone operator theory to the non-Euclidean finite-dimensional spaces $\ell_1$ and $\ell_{\infty}$. We first establish properties of mappings which are monotone with respect to the non-Euclidean norms $\ell_1$ or $\ell_{\infty}$. In analogy with their Euclidean counterparts, mappings which are monotone with respect to a non-Euclidean norm are amenable to numerous algorithms for computing their zeros. We demonstrate that several classic iterative methods for computing zeros of monotone operators are directly applicable in the non-Euclidean framework. We present a case-study in the equilibrium computation of recurrent neural networks and demonstrate that casting the computation as a suitable operator splitting problem improves convergence rates. △ Less

Submitted 18 July, 2022; v1 submitted 4 April, 2022; originally announced April 2022.

Comparative Analysis of Interval Reachability for Robust Implicit and Feedforward Neural Networks

Authors: Alexander Davydov, Saber Jafarpour, Matthew Abate, Francesco Bullo, Samuel Coogan

Abstract: We use interval reachability analysis to obtain robustness guarantees for implicit neural networks (INNs). INNs are a class of implicit learning models that use implicit equations as layers and have been shown to exhibit several notable benefits over traditional deep neural networks. We first establish that tight inclusion functions of neural networks, which provide the tightest rectangular over-a… ▽ More We use interval reachability analysis to obtain robustness guarantees for implicit neural networks (INNs). INNs are a class of implicit learning models that use implicit equations as layers and have been shown to exhibit several notable benefits over traditional deep neural networks. We first establish that tight inclusion functions of neural networks, which provide the tightest rectangular over-approximation of an input-output map, lead to sharper robustness guarantees than the well-studied robustness measures of local Lipschitz constants. Like Lipschitz constants, tight inclusions functions are computationally challenging to obtain, and we thus propose using mixed monotonicity and contraction theory to obtain computationally efficient estimates of tight inclusion functions for INNs. We show that our approach performs at least as well as, and generally better than, applying state-of-the-art interval bound propagation methods to INNs. We design a novel optimization problem for training robust INNs and we provide empirical evidence that suitably-trained INNs can be more robust than comparably-trained feedforward networks. △ Less

Submitted 31 March, 2022; originally announced April 2022.

Resilience of Input Metering in Dynamic Flow Networks

Authors: Saber Jafarpour, Samuel Coogan

Abstract: In this paper, we study robustness of input metering policies in dynamic flow networks in the presence of transient disturbances and attacks. We consider a compartmental model for dynamic flow networks with a First-In-First-Out (FIFO) routing rule as found in, e.g., transportation networks. We model the effect of the transient disturbance as an abrupt change to the state of the network and use the… ▽ More In this paper, we study robustness of input metering policies in dynamic flow networks in the presence of transient disturbances and attacks. We consider a compartmental model for dynamic flow networks with a First-In-First-Out (FIFO) routing rule as found in, e.g., transportation networks. We model the effect of the transient disturbance as an abrupt change to the state of the network and use the notion of the region of attraction to measure the resilience of the network to these changes. For constant and periodic input metering, we introduce the notion of monotone-invariant points to establish inner-estimates for the regions of attraction of free-flow equilibrium points and free-flow periodic orbits using monotone systems theory. These results are applicable to, e.g., networks with cycles, which have not been considered in prior literature on dynamic flow networks with FIFO routing. Finally, we propose two approaches for finding suitable monotone-invariant points in the flow networks with FIFO rules. △ Less

Submitted 14 March, 2022; originally announced March 2022.

Multistability and anomalies in oscillator models of lossy power grids

Authors: Robin Delabays, Saber Jafarpour, Francesco Bullo

Abstract: The analysis of dissipatively coupled oscillators is challenging and highly relevant in power grids. Standard mathematical methods are not applicable, due to the lack of network symmetry induced by dissipative couplings. Here we demonstrate a close correspondence between stable synchronous states in dissipatively coupled oscillators, and the {winding partition} of their state space, a geometric no… ▽ More The analysis of dissipatively coupled oscillators is challenging and highly relevant in power grids. Standard mathematical methods are not applicable, due to the lack of network symmetry induced by dissipative couplings. Here we demonstrate a close correspondence between stable synchronous states in dissipatively coupled oscillators, and the {winding partition} of their state space, a geometric notion induced by the network topology. Leveraging this winding partition, we accompany this article with an algorithms to compute all synchronous solutions of complex networks of dissipatively coupled oscillators. These geometric and computational tools allow us to identify anomalous behaviors of lossy networked systems. Counterintuitively, we show that loop flows and dissipation can increase the system's transfer capacity, and that dissipation can promote multistability. We apply our geometric framework to compute power flows on the IEEE RTS-96 test system, where we identify two high voltage solutions with distinct loop flows. △ Less

Submitted 6 September, 2022; v1 submitted 4 February, 2022; originally announced February 2022.

Comments: Main text: 14 pages, 5 figures. Supplementary information: 6 pages, 2 figures

Journal ref: Nat. Commun. 13, 5238 (2022)

Minimax Flow over Acyclic Networks: Distributed Algorithms and Microgrid Application

Authors: Marco Coraggio, Saber Jafarpour, Francesco Bullo, Mario di Bernardo

Abstract: Given a flow network with variable suppliers and fixed consumers, the minimax flow problem consists in minimizing the maximum flow between nodes, subject to flow conservation and capacity constraints. We solve this problem over acyclic graphs in a distributed manner by showing that it can be recast as a consensus problem between the maximum downstream flows, which we define here for the first time… ▽ More Given a flow network with variable suppliers and fixed consumers, the minimax flow problem consists in minimizing the maximum flow between nodes, subject to flow conservation and capacity constraints. We solve this problem over acyclic graphs in a distributed manner by showing that it can be recast as a consensus problem between the maximum downstream flows, which we define here for the first time. Additionally, we present a distributed algorithm to estimate these quantities. Finally, exploiting our theoretical results, we design an online distributed controller to prevent overcurrent in microgrids consisting of loads and droop-controlled inverters. Our results are validated numerically on the CIGRE benchmark microgrid. △ Less

Submitted 9 July, 2022; v1 submitted 10 January, 2022; originally announced January 2022.

Robustness Certificates for Implicit Neural Networks: A Mixed Monotone Contractive Approach

Authors: Saber Jafarpour, Matthew Abate, Alexander Davydov, Francesco Bullo, Samuel Coogan

Abstract: Implicit neural networks are a general class of learning models that replace the layers in traditional feedforward models with implicit algebraic equations. Compared to traditional learning models, implicit networks offer competitive performance and reduced memory consumption. However, they can remain brittle with respect to input adversarial perturbations. This paper proposes a theoretical and… ▽ More Implicit neural networks are a general class of learning models that replace the layers in traditional feedforward models with implicit algebraic equations. Compared to traditional learning models, implicit networks offer competitive performance and reduced memory consumption. However, they can remain brittle with respect to input adversarial perturbations. This paper proposes a theoretical and computational framework for robustness verification of implicit neural networks; our framework blends together mixed monotone systems theory and contraction theory. First, given an implicit neural network, we introduce a related embedded network and show that, given an $\ell_\infty$-norm box constraint on the input, the embedded network provides an $\ell_\infty$-norm box overapproximation for the output of the given network. Second, using $\ell_{\infty}$-matrix measures, we propose sufficient conditions for well-posedness of both the original and embedded system and design an iterative algorithm to compute the $\ell_{\infty}$-norm box robustness margins for reachability and classification problems. Third, of independent value, we propose a novel relative classifier variable that leads to tighter bounds on the certified adversarial robustness in classification problems. Finally, we perform numerical simulations on a Non-Euclidean Monotone Operator Network (NEMON) trained on the MNIST dataset. In these simulations, we compare the accuracy and run time of our mixed monotone contractive approach with the existing robustness verification approaches in the literature for estimating the certified adversarial robustness. △ Less

Submitted 9 December, 2021; originally announced December 2021.

From Contraction Theory to Fixed Point Algorithms on Riemannian and Non-Euclidean Spaces

Authors: Francesco Bullo, Pedro Cisneros-Velarde, Alexander Davydov, Saber Jafarpour

Abstract: The design of fixed point algorithms is at the heart of monotone operator theory, convex analysis, and of many modern optimization problems arising in machine learning and control. This tutorial reviews recent advances in understanding the relationship between Demidovich conditions, one-sided Lipschitz conditions, and contractivity theorems. We review the standard contraction theory on Euclidean s… ▽ More The design of fixed point algorithms is at the heart of monotone operator theory, convex analysis, and of many modern optimization problems arising in machine learning and control. This tutorial reviews recent advances in understanding the relationship between Demidovich conditions, one-sided Lipschitz conditions, and contractivity theorems. We review the standard contraction theory on Euclidean spaces as well as little-known results for Riemannian manifolds. Special emphasis is placed on the setting of non-Euclidean norms and the recently introduced weak pairings for the $\ell_1$ and $\ell_\infty$ norms. We highlight recent results on explicit and implicit fixed point schemes for non-Euclidean contracting systems. △ Less

Submitted 7 October, 2021; originally announced October 2021.

Comments: Paper in the invited tutorial session "Contraction Theory for Machine Learning" at 60th IEEE Conference on Decision and Control, 2021

Robust Implicit Networks via Non-Euclidean Contractions

Authors: Saber Jafarpour, Alexander Davydov, Anton V. Proskurnikov, Francesco Bullo

Abstract: Implicit neural networks, a.k.a., deep equilibrium networks, are a class of implicit-depth learning models where function evaluation is performed by solving a fixed point equation. They generalize classic feedforward models and are equivalent to infinite-depth weight-tied feedforward networks. While implicit models show improved accuracy and significant reduction in memory consumption, they can su… ▽ More Implicit neural networks, a.k.a., deep equilibrium networks, are a class of implicit-depth learning models where function evaluation is performed by solving a fixed point equation. They generalize classic feedforward models and are equivalent to infinite-depth weight-tied feedforward networks. While implicit models show improved accuracy and significant reduction in memory consumption, they can suffer from ill-posedness and convergence instability. This paper provides a new framework, which we call Non-Euclidean Monotone Operator Network (NEMON), to design well-posed and robust implicit neural networks based upon contraction theory for the non-Euclidean norm $\ell_{\infty}$. Our framework includes (i) a novel condition for well-posedness based on one-sided Lipschitz constants, (ii) an average iteration for computing fixed-points, and (iii) explicit estimates on input-output Lipschitz constants. Additionally, we design a training problem with the well-posedness condition and the average iteration as constraints and, to achieve robust models, with the input-output Lipschitz constant as a regularizer. Our $\ell_{\infty}$ well-posedness condition leads to a larger polytopic training search space than existing conditions and our average iteration enjoys accelerated convergence. Finally, we evaluate our framework in image classification through the MNIST and the CIFAR-10 datasets. Our numerical results demonstrate improved accuracy and robustness of the implicit models with smaller input-output Lipschitz bounds. Code is available at https://github.com/davydovalexander/Non-Euclidean_Mon_Op_Net. △ Less

Submitted 25 January, 2022; v1 submitted 6 June, 2021; originally announced June 2021.

Non-Euclidean Contraction Theory for Monotone and Positive Systems

Authors: Saber Jafarpour, Alexander Davydov, Francesco Bullo

Abstract: In this note we study contractivity of monotone systems and exponential convergence of positive systems using non-Euclidean norms. We first introduce the notion of conic matrix measure as a framework to study stability of monotone and positive systems. We study properties of the conic matrix measures and investigate their connection with weak pairings and standard matrix measures. Using conic matr… ▽ More In this note we study contractivity of monotone systems and exponential convergence of positive systems using non-Euclidean norms. We first introduce the notion of conic matrix measure as a framework to study stability of monotone and positive systems. We study properties of the conic matrix measures and investigate their connection with weak pairings and standard matrix measures. Using conic matrix measures and weak pairings, we characterize contractivity and incremental stability of monotone systems with respect to non-Euclidean norms. Moreover, we use conic matrix measures to provide sufficient conditions for exponential convergence of positive systems to their equilibria. We show that our framework leads to novel results on (i) the contractivity of excitatory Hopfield neural networks, and (ii) the stability of interconnected systems using non-monotone positive comparison systems. △ Less

Submitted 21 August, 2022; v1 submitted 3 April, 2021; originally announced April 2021.

Non-Euclidean Contraction Theory for Robust Nonlinear Stability

Authors: Alexander Davydov, Saber Jafarpour, Francesco Bullo

Abstract: We study necessary and sufficient conditions for contraction and incremental stability of dynamical systems with respect to non-Euclidean norms. First, we introduce weak pairings as a framework to study contractivity with respect to arbitrary norms, and characterize their properties. We introduce and study the sign and max pairings for the $\ell_1$ and $\ell_\infty$ norms, respectively. Using weak… ▽ More We study necessary and sufficient conditions for contraction and incremental stability of dynamical systems with respect to non-Euclidean norms. First, we introduce weak pairings as a framework to study contractivity with respect to arbitrary norms, and characterize their properties. We introduce and study the sign and max pairings for the $\ell_1$ and $\ell_\infty$ norms, respectively. Using weak pairings, we establish five equivalent characterizations for contraction, including the one-sided Lipschitz condition for the vector field as well as matrix measure and Demidovich conditions for the corresponding Jacobian. Third, we extend our contraction framework in two directions: we prove equivalences for contraction of continuous vector fields and we formalize the weaker notion of equilibrium contraction, which ensures exponential convergence to an equilibrium. Finally, as an application, we provide (i) incremental input-to-state stability and finite input-state gain properties for contracting systems, and (ii) a general theorem about the Lipschitz interconnection of contracting systems, whereby the Hurwitzness of a gain matrix implies the contractivity of the interconnected system. △ Less

Submitted 1 August, 2022; v1 submitted 22 March, 2021; originally announced March 2021.

Contraction Theory for Dynamical Systems on Hilbert Spaces

Authors: Pedro Cisneros-Velarde, Saber Jafarpour, Francesco Bullo

Abstract: Contraction theory for dynamical systems on Euclidean spaces is well-established. For contractive (resp. semi-contractive) systems, the distance (resp. semi-distance) between any two trajectories decreases exponentially fast. For partially contractive systems, each trajectory converges exponentially fast to an invariant subspace. In this note, we develop contraction theory on Hilbert spaces. Fir… ▽ More Contraction theory for dynamical systems on Euclidean spaces is well-established. For contractive (resp. semi-contractive) systems, the distance (resp. semi-distance) between any two trajectories decreases exponentially fast. For partially contractive systems, each trajectory converges exponentially fast to an invariant subspace. In this note, we develop contraction theory on Hilbert spaces. First, we provide a novel integral condition for contractivity, and for time-invariant systems, we establish the existence of a unique globally exponentially stable equilibrium. Second, we introduce the notions of partial and semi-contraction and we provide various sufficient conditions for time-varying and time-invariant systems. Finally, we apply the theory on a classic reaction-diffusion system. △ Less

Submitted 4 June, 2021; v1 submitted 2 October, 2020; originally announced October 2020.

MSC Class: 34G20; 93D23 (Primary) 93D23; 37N99; 93D20 (Secondary)

Weak and Semi-Contraction for Network Systems and Diffusively-Coupled Oscillators

Authors: Saber Jafarpour, Pedro Cisneros-Velarde, Francesco Bullo

Abstract: We develop two generalizations of contraction theory, namely, semi-contraction and weak-contraction theory. First, using the notion of semi-norm, we propose a geometric framework for semi-contraction theory. We introduce matrix semi-measures and characterize their properties. We show that the spectral abscissa of a matrix is the infimum over weighted semi-measures. For dynamical systems, we use th… ▽ More We develop two generalizations of contraction theory, namely, semi-contraction and weak-contraction theory. First, using the notion of semi-norm, we propose a geometric framework for semi-contraction theory. We introduce matrix semi-measures and characterize their properties. We show that the spectral abscissa of a matrix is the infimum over weighted semi-measures. For dynamical systems, we use the semi-measure of their Jacobian to characterize the contractivity properties of their trajectories. Second, for weakly contracting systems, we prove a dichotomy for the asymptotic behavior of their trajectories and novel sufficient conditions for convergence to an equilibrium. Third, we show that every trajectory of a doubly-contracting system, i.e., a system that is both weakly and semi-contracting, converges to an equilibrium point. Finally, we apply our results to various important network systems including affine averaging and affine flow systems, continuous-time distributed primal-dual algorithms, and networks of diffusively-coupled dynamical systems. For diffusively-coupled systems, the semi-contraction theory leads to a sufficient condition for synchronization that is sharper, in general, than previously-known tests. △ Less

Submitted 4 October, 2020; v1 submitted 19 May, 2020; originally announced May 2020.

Topology Inference with Multivariate Cumulants: The Möbius Inference Algorithm

Authors: Kevin D. Smith, Saber Jafarpour, Ananthram Swami, Francesco Bullo

Abstract: Many tasks regarding the monitoring, management, and design of communication networks rely on knowledge of the routing topology. However, the standard approach to topology mapping--namely, active probing with traceroutes--relies on cooperation from increasingly non-cooperative routers, leading to missing information. Network tomography, which uses end-to-end measurements of additive link metrics (… ▽ More Many tasks regarding the monitoring, management, and design of communication networks rely on knowledge of the routing topology. However, the standard approach to topology mapping--namely, active probing with traceroutes--relies on cooperation from increasingly non-cooperative routers, leading to missing information. Network tomography, which uses end-to-end measurements of additive link metrics (like delays or log packet loss rates) across monitor paths, is a possible remedy. Network tomography does not require that routers cooperate with traceroute probes, and it has already been used to infer the structure of multicast trees. This paper goes a step further. We provide a tomographic method to infer the underlying routing topology of an arbitrary set of monitor paths using the joint distribution of end-to-end measurements, without making any assumptions on routing behavior. Our approach, called the Möbius Inference Algorithm (MIA), uses cumulants of this distribution to quantify high-order interactions among monitor paths, and it applies Möbius inversion to "disentangle" these interactions. In addition to MIA, we provide a more practical variant called Sparse Möbius Inference, which uses various sparsity heuristics to reduce the number and order of cumulants required to be estimated. We show the viability of our approach using synthetic case studies based on real-world ISP topologies. △ Less

Submitted 18 June, 2021; v1 submitted 16 May, 2020; originally announced May 2020.

Distributed and time-varying primal-dual dynamics via contraction analysis

Authors: Pedro Cisneros-Velarde, Saber Jafarpour, Francesco Bullo

Abstract: In this note, we provide an overarching analysis of primal-dual dynamics associated to linear equality-constrained optimization problems using contraction analysis. For the well-known standard version of the problem: we establish convergence under convexity and the contracting rate under strong convexity. Then, for a canonical distributed optimization problem, we use partial contractivity to estab… ▽ More In this note, we provide an overarching analysis of primal-dual dynamics associated to linear equality-constrained optimization problems using contraction analysis. For the well-known standard version of the problem: we establish convergence under convexity and the contracting rate under strong convexity. Then, for a canonical distributed optimization problem, we use partial contractivity to establish global exponential convergence of its primal-dual dynamics. As an application, we propose a new distributed solver for the least-squares problem with the same convergence guarantees. Finally, for time-varying versions of both centralized and distributed primal-dual dynamics, we exploit their contractive nature to establish bounds on their tracking error. To support our analyses, we introduce novel results on contraction theory. △ Less

Submitted 19 June, 2021; v1 submitted 27 March, 2020; originally announced March 2020.

MSC Class: 34H05 (Primary); 34D23; 90C25; 93C95 (Secondary) ACM Class: G.1.7; G.1.6

Transient Stability of Droop-Controlled Inverter Networks with Operating Constraints

Authors: Kevin D. Smith, Saber Jafarpour, Francesco Bullo

Abstract: Due to the rise of distributed energy resources, the control of networks of grid-forming inverters is now a pressing issue for power system operation. Droop control is a popular control strategy in the literature for frequency control of these inverters. In this paper, we analyze transient stability in droop-controlled inverter networks that are subject to multiple operating constraints. Using a p… ▽ More Due to the rise of distributed energy resources, the control of networks of grid-forming inverters is now a pressing issue for power system operation. Droop control is a popular control strategy in the literature for frequency control of these inverters. In this paper, we analyze transient stability in droop-controlled inverter networks that are subject to multiple operating constraints. Using a physically-meaningful Lyapunov-like function, we provide two sets of criteria (one mathematical and one computational) to certify that a post-fault trajectory achieves frequency synchronization while respecting operating constraints. We show how to obtain less-conservative transient stability conditions by incorporating information from loop flows, i.e., net flows of active power around cycles in the network. Finally, we use these conditions to quantify the scale of parameter disturbances to which the network is robust. We illustrate our results with numerical case studies of the IEEE 24-bus system. △ Less

Submitted 6 September, 2020; v1 submitted 11 July, 2019; originally announced July 2019.

Graph-Theoretic Stability Conditions for Metzler Matrices and Monotone Systems

Authors: Xiaoming Duan, Saber Jafarpour, Francesco Bullo

Abstract: This paper studies the graph-theoretic conditions for stability of positive monotone systems. Using concepts from input-to-state stability and network small-gain theory, we first establish necessary and sufficient conditions for the stability of linear positive systems described by Metzler matrices. Specifically, we derive two sets of stability conditions based on two forms of input-to-state stabi… ▽ More This paper studies the graph-theoretic conditions for stability of positive monotone systems. Using concepts from input-to-state stability and network small-gain theory, we first establish necessary and sufficient conditions for the stability of linear positive systems described by Metzler matrices. Specifically, we derive two sets of stability conditions based on two forms of input-to-state stability gains for Metzler systems, namely max-interconnection gains and sum-interconnection gains. Based on the max-interconnection gains, we show that the cyclic small-gain theorem becomes necessary and sufficient for the stability of Metzler systems; based on the sum-interconnection gains, we obtain novel graph-theoretic conditions for the stability of Metzler systems. All these conditions highlight the role of cycles in the interconnection graph and unveil how the structural properties of the graph affect stability. Finally, we extend our results to the nonlinear monotone system and obtain similar sufficient conditions for global asymptotic stability. △ Less

Submitted 22 May, 2020; v1 submitted 14 May, 2019; originally announced May 2019.

Singular Perturbation and Small-signal Stability for Inverter Networks

Authors: Saber Jafarpour, Victor Purba, Sairaj V. Dhople, Brian Johnson, Francesco Bullo

Abstract: This paper examines small-signal stability of electrical networks composed dominantly of three-phase grid-following inverters. We show that the mere existence of a high-voltage power flow solution does not necessarily imply small-signal stability; this motivates us to develop a framework for stability analysis that systematically acknowledges inverter dynamics. We identify a suitable time-scale de… ▽ More This paper examines small-signal stability of electrical networks composed dominantly of three-phase grid-following inverters. We show that the mere existence of a high-voltage power flow solution does not necessarily imply small-signal stability; this motivates us to develop a framework for stability analysis that systematically acknowledges inverter dynamics. We identify a suitable time-scale decomposition for the inverter dynamics, and using singular perturbation theory, obtain an analytic sufficient condition to verify small-signal stability. Compared to the alternative of performing an eigenvalue analysis of the full-order network dynamics, our analytic sufficient condition reduces computational complexity and yields insights on the role of network topology and constitution as well as inverter-filter and control parameters on small-signal stability. Numerical simulations for a radial network validate the approach and illustrate the efficiency of our analytic conditions for designing and monitoring grid-tied inverter networks. △ Less

Submitted 20 May, 2020; v1 submitted 7 February, 2019; originally announced February 2019.

Flow and Elastic Networks on the $n$-torus: Geometry, Analysis, and Computation

Authors: Saber Jafarpour, Elizabeth Y. Huang, Kevin D. Smith, Francesco Bullo

Abstract: Networks with phase-valued nodal variables are central in modeling several important societal and physical systems, including power grids, biological systems, and coupled oscillator networks. One of the distinctive features of phase-valued networks is the existence of multiple operating conditions corresponding to critical points of an energy function or feasible flows of a balance equation. For n… ▽ More Networks with phase-valued nodal variables are central in modeling several important societal and physical systems, including power grids, biological systems, and coupled oscillator networks. One of the distinctive features of phase-valued networks is the existence of multiple operating conditions corresponding to critical points of an energy function or feasible flows of a balance equation. For networks with phase-valued states, it is not yet fully understood how many operating conditions exist, how to characterize them, and how to compute them efficiently. A deeper understanding of feasible operating conditions, including their dependence upon network structures, may lead to more reliable and efficient network systems. This paper introduces flow and elastic network problems on the $n$-torus and provides a rigorous and comprehensive framework for their study. Based on a monotonicity assumption, this framework localizes the solutions, bounds their number, and leads to an algorithm to compute them. Our analysis is based on a novel winding partition of the $n$-torus into winding cells, induced by Kirchhoff's angle law for undirected graphs. The winding partition has several useful properties, including notably that, each winding cell contains at most one solution. The proposed algorithm is based on a novel contraction mapping and is guaranteed to compute all solutions. Finally, we apply our results to numerically study the active power flow equations in several test cases and estimate power capacity and congestion of a power network. △ Less

Submitted 12 September, 2020; v1 submitted 30 January, 2019; originally announced January 2019.

Synchronization of Kuramoto Oscillators: Inverse Taylor Expansions

Authors: Saber Jafarpour, Elizabeth Y. Huang, Francesco Bullo

Abstract: Synchronization in networks of coupled oscillators is a widely studied topic with extensive scientific and engineering applications. In this paper, we study the frequency synchronization problem for networks of Kuramoto oscillators with arbitrary topology and heterogeneous edge weights. We propose a novel equivalent transcription for the equilibrium synchronization equation. Using this transcripti… ▽ More Synchronization in networks of coupled oscillators is a widely studied topic with extensive scientific and engineering applications. In this paper, we study the frequency synchronization problem for networks of Kuramoto oscillators with arbitrary topology and heterogeneous edge weights. We propose a novel equivalent transcription for the equilibrium synchronization equation. Using this transcription, we develop a power series expansion to compute the synchronized solution of the Kuramoto model as well as a sufficient condition for the strong convergence of this series expansion. Truncating the power series provides (i) an efficient approximation scheme for computing the synchronized solution, and (ii) a simple-to-check, statistically-correct hierarchy of increasingly accurate synchronization tests. This hierarchy of tests provides a theoretical foundation for and generalizes the best-known approximate synchronization test in the literature. Our numerical experiments illustrate the accuracy and the computational efficiency of the truncated series approximation compared to existing iterative methods and existing synchronization tests. △ Less

Submitted 23 September, 2018; originally announced September 2018.

Synchronization of Coupled Oscillators: The Taylor Expansion of the Inverse Kuramoto Map

Authors: Elizabeth Y. Huang, Saber Jafarpour, Francesco Bullo

Abstract: Synchronization in the networks of coupled oscillators is a widely studied topic in different areas. It is well-known that synchronization occurs if the connectivity of the network dominates heterogeneity of the oscillators. Despite extensive study on this topic, the quest for sharp closed-form synchronization tests is still in vain. In this paper, we present an algorithm for finding the Taylor ex… ▽ More Synchronization in the networks of coupled oscillators is a widely studied topic in different areas. It is well-known that synchronization occurs if the connectivity of the network dominates heterogeneity of the oscillators. Despite extensive study on this topic, the quest for sharp closed-form synchronization tests is still in vain. In this paper, we present an algorithm for finding the Taylor expansion of the inverse Kuramoto map. We show that this Taylor series can be used to obtain a hierarchy of increasingly accurate approximate tests with low computational complexity. These approximate tests are then used to estimate the threshold of synchronization as well as the position of the synchronization manifold of the network. △ Less

Submitted 19 September, 2018; v1 submitted 22 March, 2018; originally announced March 2018.

Synchronization of Kuramoto Oscillators via Cutset Projections

Authors: Saber Jafarpour, Francesco Bullo

Abstract: Synchronization in coupled oscillators networks is a remarkable phenomenon of relevance in numerous fields. For Kuramoto oscillators the loss of synchronization is determined by a trade-off between coupling strength and oscillator heterogeneity. Despite extensive prior work, the existing sufficient conditions for synchronization are either very conservative or heuristic and approximate. Using a no… ▽ More Synchronization in coupled oscillators networks is a remarkable phenomenon of relevance in numerous fields. For Kuramoto oscillators the loss of synchronization is determined by a trade-off between coupling strength and oscillator heterogeneity. Despite extensive prior work, the existing sufficient conditions for synchronization are either very conservative or heuristic and approximate. Using a novel cutset projection operator, we propose a new family of sufficient synchronization conditions; these conditions rigorously identify the correct functional form of the trade-off between coupling strength and oscillator heterogeneity. To overcome the need to solve a nonconvex optimization problem, we then provide two explicit bounding methods, thereby obtaining (i) the best-known sufficient condition for unweighted graphs based on the 2-norm, and (ii) the first-known generally-applicable sufficient condition based on the $\infty$-norm. We conclude with a comparative study of our novel $\infty$-norm condition for specific topologies and IEEE test cases; for most IEEE test cases our new sufficient condition is one to two orders of magnitude more accurate than previous rigorous tests. △ Less

Submitted 26 July, 2018; v1 submitted 10 November, 2017; originally announced November 2017.

On small-time local controllability

Authors: Saber Jafarpour

Abstract: In this paper, we study small-time local controllability of real analytic control-affine systems under small perturbations of their vector fields. Consider a real analytic control system $\mathcal{X}$ which is small-time locally controllable and whose reachable sets shrink with the polynomial rate of order $N$ with respect to time. We will prove a general theorem which states that any real analyti… ▽ More In this paper, we study small-time local controllability of real analytic control-affine systems under small perturbations of their vector fields. Consider a real analytic control system $\mathcal{X}$ which is small-time locally controllable and whose reachable sets shrink with the polynomial rate of order $N$ with respect to time. We will prove a general theorem which states that any real analytic control-affine system whose vector fields are perturbations of the vector fields of $\mathcal{X}$ with polynomials of order higher than $N$ is again small-time locally controllable. In particular, we show that this result connects two long-standing open conjectures about small-time local controllability of systems. △ Less

Submitted 18 November, 2019; v1 submitted 8 April, 2016; originally announced April 2016.

Comments: 22 pages

Local and global holomorphic extensions of time-varying real analytic vector fields

Authors: Saber Jafarpour

Abstract: In this paper, we consider time-varying real analytic vector fields as curves on the space of real analytic vector fields. Using a suitable topology on the space of real analytic vector fields, we study and characterize different properties of time-varying real analytic vector fields. We study holomorphic extensions of time-varying real analytic vector fields and show that under suitable integrabi… ▽ More In this paper, we consider time-varying real analytic vector fields as curves on the space of real analytic vector fields. Using a suitable topology on the space of real analytic vector fields, we study and characterize different properties of time-varying real analytic vector fields. We study holomorphic extensions of time-varying real analytic vector fields and show that under suitable integrability conditions, a time-varying real analytic vector field on a manifold can be extended to a time-varying holomorphic vector field on a neighbourhood of that manifold. Moreover, we develop an operator setting, where the nonlinear differential equation governing the flow of a time-varying real analytic vector field can be considered as a linear differential equation on an infinite dimensional locally convex vector space. Using the holomorphic extension results, we show that the integrability of the time-varying vector field ensures the convergence of the sequence of Picard iterations for this linear differential equation. This gives us a series representation for the flow of an integrable time-varying real analytic vector field. We also define the exponential map between integrable time-varying real analytic vector fields and their flows. Using the holomorphic extensions of time-varying real analytic vector fields, we show that the exponential map is sequentially continuous. △ Less

Submitted 8 April, 2016; originally announced April 2016.

Linear Inverse Problems with Norm and Sparsity Constraints

Authors: Volkan Cevher, Sina Jafarpour, Anastasios Kyrillidis

Abstract: We describe two nonconventional algorithms for linear regression, called GAME and CLASH. The salient characteristics of these approaches is that they exploit the convex $\ell_1$-ball and non-convex $\ell_0$-sparsity constraints jointly in sparse recovery. To establish the theoretical approximation guarantees of GAME and CLASH, we cover an interesting range of topics from game theory, convex and co… ▽ More We describe two nonconventional algorithms for linear regression, called GAME and CLASH. The salient characteristics of these approaches is that they exploit the convex $\ell_1$-ball and non-convex $\ell_0$-sparsity constraints jointly in sparse recovery. To establish the theoretical approximation guarantees of GAME and CLASH, we cover an interesting range of topics from game theory, convex and combinatorial optimization. We illustrate that these approaches lead to improved theoretical guarantees and empirical performance beyond convex and non-convex solvers alone. △ Less

Submitted 19 July, 2015; originally announced July 2015.

Comments: 21 pages, authors in alphabetical order

Mathematical models for geometric control theory

Authors: Saber Jafarpour, Andrew D. Lewis

Abstract: Just as an explicit parameterisation of system dynamics by state, i.e., a choice of coordinates, can impede the identification of general structure, so it is too with an explicit parameterisation of system dynamics by control. However, such explicit and fixed parameterisation by control is commonplace in control theory, leading to definitions, methodologies, and results that depend in unexpected w… ▽ More Just as an explicit parameterisation of system dynamics by state, i.e., a choice of coordinates, can impede the identification of general structure, so it is too with an explicit parameterisation of system dynamics by control. However, such explicit and fixed parameterisation by control is commonplace in control theory, leading to definitions, methodologies, and results that depend in unexpected ways on control parameterisation. In this paper a framework is presented for modelling systems in geometric control theory in a manner that does not make any choice of parameterisation by control; the systems are called "tautological control systems." For the framework to be coherent, it relies in a fundamental way on topologies for spaces of vector fields. As such, classes of systems are considered possessing a variety of degrees of regularity: finitely differentiable; Lipschitz; smooth; real analytic. In each case, explicit geometric seminorms are provided for the topologies of spaces of vector fields that enable straightforward descriptions of time-varying vector fields and control systems. As part of the development, theorems are proved for regular (including real analytic) dependence on initial conditions of flows of vector fields depending measurably on time. Classes of "ordinary" control systems are characterised that interact with the regularity under consideration in a comprehensive way. In this framework, for example, the statement that "a smooth or real analytic control-affine system is a smooth or real analytic control system" becomes a theorem. Correspondences between ordinary control systems and tautological control systems are carefully examined, and trajectory correspondence between the two classes is proved for control-affine systems and for systems with general control dependence when the control set is compact. △ Less

Submitted 13 June, 2014; v1 submitted 23 December, 2013; originally announced December 2013.

Comments: 203 pages, 182 references

MSC Class: 93A30 (Primary) 32C05; 34A12; 34A60; 46E10; 93B17; 93B18; 93B27; 93B99 (Secondary)

Quasi-Optimal Network Utility Maximization for Scalable Video Streaming

Authors: Mohammad Sadegh Talebi, Ahmad Khonsari, Mohammad Hassan Hajiesmaili, Sina Jafarpour

Abstract: This paper addresses rate control for transmission of scalable video streams via Network Utility Maximization (NUM) formulation. Due to stringent QoS requirements of video streams and specific characterization of utility experienced by end-users, one has to solve nonconvex and even nonsmooth NUM formulation for such streams, where dual methods often prove incompetent. Convexification plays an impo… ▽ More This paper addresses rate control for transmission of scalable video streams via Network Utility Maximization (NUM) formulation. Due to stringent QoS requirements of video streams and specific characterization of utility experienced by end-users, one has to solve nonconvex and even nonsmooth NUM formulation for such streams, where dual methods often prove incompetent. Convexification plays an important role in this work as it permits the use of existing dual methods to solve an approximate to the NUM problem iteratively and distributively. Hence, to tackle the nonsmoothness and nonconvexity, we aim at reformulating the NUM problem through approximation and transformation of the ideal discretely adaptive utility function for scalable video streams. The reformulated problem is shown to be a D.C. (Difference of Convex) problem. We leveraged Sequential Convex Programming (SCP) approach to replace the nonconvex D.C. problem by a sequence of convex problems that aim to approximate the original D.C. problem. We then solve each convex problem produced by SCP approach using existing dual methods. This procedure is the essence of two distributed iterative rate control algorithms proposed in this paper, for which one can show the convergence to a locally optimal point of the nonconvex D.C. problem and equivalently to a locally optimal point of an approximate to the original nonconvex problem. Our experimental results show that the proposed rate control algorithms converge with tractable convergence behavior. △ Less

Submitted 17 August, 2012; v1 submitted 13 February, 2011; originally announced February 2011.

Comments: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible

Performance bounds for expander-based compressed sensing in Poisson noise

Authors: Maxim Raginsky, Sina Jafarpour, Zachary Harmany, Roummel Marcia, Rebecca Willett, Robert Calderbank

Abstract: This paper provides performance bounds for compressed sensing in the presence of Poisson noise using expander graphs. The Poisson noise model is appropriate for a variety of applications, including low-light imaging and digital streaming, where the signal-independent and/or bounded noise models used in the compressed sensing literature are no longer applicable. In this paper, we develop a novel se… ▽ More This paper provides performance bounds for compressed sensing in the presence of Poisson noise using expander graphs. The Poisson noise model is appropriate for a variety of applications, including low-light imaging and digital streaming, where the signal-independent and/or bounded noise models used in the compressed sensing literature are no longer applicable. In this paper, we develop a novel sensing paradigm based on expander graphs and propose a MAP algorithm for recovering sparse or compressible signals from Poisson observations. The geometry of the expander graphs and the positivity of the corresponding sensing matrices play a crucial role in establishing the bounds on the signal reconstruction error of the proposed algorithm. We support our results with experimental demonstrations of reconstructing average packet arrival rates and instantaneous packet counts at a router in a communication network, where the arrivals of packets in each flow follow a Poisson process. △ Less

Submitted 20 May, 2011; v1 submitted 14 July, 2010; originally announced July 2010.

Comments: revised version; accepted to IEEE Transactions on Signal Processing

Why Gabor Frames? Two Fundamental Measures of Coherence and Their Role in Model Selection

Authors: Waheed U. Bajwa, Robert Calderbank, Sina Jafarpour

Abstract: This paper studies non-asymptotic model selection for the general case of arbitrary design matrices and arbitrary nonzero entries of the signal. In this regard, it generalizes the notion of incoherence in the existing literature on model selection and introduces two fundamental measures of coherence---termed as the worst-case coherence and the average coherence---among the columns of a design matr… ▽ More This paper studies non-asymptotic model selection for the general case of arbitrary design matrices and arbitrary nonzero entries of the signal. In this regard, it generalizes the notion of incoherence in the existing literature on model selection and introduces two fundamental measures of coherence---termed as the worst-case coherence and the average coherence---among the columns of a design matrix. It utilizes these two measures of coherence to provide an in-depth analysis of a simple, model-order agnostic one-step thresholding (OST) algorithm for model selection and proves that OST is feasible for exact as well as partial model selection as long as the design matrix obeys an easily verifiable property. One of the key insights offered by the ensuing analysis in this regard is that OST can successfully carry out model selection even when methods based on convex optimization such as the lasso fail due to the rank deficiency of the submatrices of the design matrix. In addition, the paper establishes that if the design matrix has reasonably small worst-case and average coherence then OST performs near-optimally when either (i) the energy of any nonzero entry of the signal is close to the average signal energy per nonzero entry or (ii) the signal-to-noise ratio in the measurement system is not too high. Finally, two other key contributions of the paper are that (i) it provides bounds on the average coherence of Gaussian matrices and Gabor frames, and (ii) it extends the results on model selection using OST to low-complexity, model-order agnostic recovery of sparse signals with arbitrary nonzero entries. △ Less

Submitted 2 July, 2010; v1 submitted 3 June, 2010; originally announced June 2010.

Comments: 31 pages, 4 figures; This paper is a full-length journal version of a shorter paper that was presented at the IEEE International Symposium on Information Theory, Austin, TX, June 2010

Journal ref: J. Commun. Netw., vol. 12, no. 4, pp. 289-307, Aug. 2010

Reed Muller Sensing Matrices and the LASSO

Authors: Robert Calderbank, Sina Jafarpour

Abstract: We construct two families of deterministic sensing matrices where the columns are obtained by exponentiating codewords in the quaternary Delsarte-Goethals code $DG(m,r)$. This method of construction results in sensing matrices with low coherence and spectral norm. The first family, which we call Delsarte-Goethals frames, are $2^m$ - dimensional tight frames with redundancy $2^{rm}$. The second fam… ▽ More We construct two families of deterministic sensing matrices where the columns are obtained by exponentiating codewords in the quaternary Delsarte-Goethals code $DG(m,r)$. This method of construction results in sensing matrices with low coherence and spectral norm. The first family, which we call Delsarte-Goethals frames, are $2^m$ - dimensional tight frames with redundancy $2^{rm}$. The second family, which we call Delsarte-Goethals sieves, are obtained by subsampling the column vectors in a Delsarte-Goethals frame. Different rows of a Delsarte-Goethals sieve may not be orthogonal, and we present an effective algorithm for identifying all pairs of non-orthogonal rows. The pairs turn out to be duplicate measurements and eliminating them leads to a tight frame. Experimental results suggest that all $DG(m,r)$ sieves with $m\leq 15$ and $r\geq2$ are tight-frames; there are no duplicate rows. For both families of sensing matrices, we measure accuracy of reconstruction (statistical 0-1 loss) and complexity (average reconstruction time) as a function of the sparsity level $k$. Our results show that DG frames and sieves outperform random Gaussian matrices in terms of noiseless and noisy signal recovery using the LASSO. △ Less

Submitted 28 April, 2010; originally announced April 2010.

Sparse Reconstruction via The Reed-Muller Sieve

Authors: Robert Calderbank, Stephen Howard, Sina Jafarpour

Abstract: This paper introduces the Reed Muller Sieve, a deterministic measurement matrix for compressed sensing. The columns of this matrix are obtained by exponentiating codewords in the quaternary second order Reed Muller code of length $N$. For $k=O(N)$, the Reed Muller Sieve improves upon prior methods for identifying the support of a $k$-sparse vector by removing the requirement that the signal entrie… ▽ More This paper introduces the Reed Muller Sieve, a deterministic measurement matrix for compressed sensing. The columns of this matrix are obtained by exponentiating codewords in the quaternary second order Reed Muller code of length $N$. For $k=O(N)$, the Reed Muller Sieve improves upon prior methods for identifying the support of a $k$-sparse vector by removing the requirement that the signal entries be independent. The Sieve also enables local detection; an algorithm is presented with complexity $N^2 \log N$ that detects the presence or absence of a signal at any given position in the data domain without explicitly reconstructing the entire signal. Reconstruction is shown to be resilient to noise in both the measurement and data domains; the $\ell_2 / \ell_2$ error bounds derived in this paper are tighter than the $\ell_2 / \ell_1$ bounds arising from random ensembles and the $\ell_1 /\ell_1$ bounds arising from expander-based ensembles. △ Less

Submitted 16 April, 2010; originally announced April 2010.

Comments: To appear in ISIT 2010

Fishing in Poisson streams: focusing on the whales, ignoring the minnows

Authors: Maxim Raginsky, Sina Jafarpour, Rebecca Willett, Robert Calderbank

Abstract: This paper describes a low-complexity approach for reconstructing average packet arrival rates and instantaneous packet counts at a router in a communication network, where the arrivals of packets in each flow follow a Poisson process. Assuming that the rate vector of this Poisson process is sparse or approximately sparse, the goal is to maintain a compressed summary of the process sample paths u… ▽ More This paper describes a low-complexity approach for reconstructing average packet arrival rates and instantaneous packet counts at a router in a communication network, where the arrivals of packets in each flow follow a Poisson process. Assuming that the rate vector of this Poisson process is sparse or approximately sparse, the goal is to maintain a compressed summary of the process sample paths using a small number of counters, such that at any time it is possible to reconstruct both the total number of packets in each flow and the underlying rate vector. We show that these tasks can be accomplished efficiently and accurately using compressed sensing with expander graphs. In particular, the compressive counts are a linear transformation of the underlying counting process by the adjacency matrix of an unbalanced expander. Such a matrix is binary and sparse, which allows for efficient incrementing when new packets arrive. We describe, analyze, and compare two methods that can be used to estimate both the current vector of total packet counts and the underlying vector of arrival rates. △ Less

Submitted 14 March, 2010; originally announced March 2010.

Comments: 6 pages, 6 pdf figures; invited paper to appear in CISS 2010

Model Selection: Two Fundamental Measures of Coherence and Their Algorithmic Significance

Authors: Waheed U. Bajwa, Robert Calderbank, Sina Jafarpour

Abstract: The problem of model selection arises in a number of contexts, such as compressed sensing, subset selection in linear regression, estimation of structures in graphical models, and signal denoising. This paper generalizes the notion of \emph{incoherence} in the existing literature on model selection and introduces two fundamental measures of coherence---termed as the worst-case coherence and the av… ▽ More The problem of model selection arises in a number of contexts, such as compressed sensing, subset selection in linear regression, estimation of structures in graphical models, and signal denoising. This paper generalizes the notion of \emph{incoherence} in the existing literature on model selection and introduces two fundamental measures of coherence---termed as the worst-case coherence and the average coherence---among the columns of a design matrix. In particular, it utilizes these two measures of coherence to provide an in-depth analysis of a simple one-step thresholding (OST) algorithm for model selection. One of the key insights offered by the ensuing analysis is that OST is feasible for model selection as long as the design matrix obeys an easily verifiable property. In addition, the paper also characterizes the model-selection performance of OST in terms of the worst-case coherence, μみゅー, and establishes that OST performs near-optimally in the low signal-to-noise ratio regime for N x C design matrices with μみゅー= O(N^{-1/2}). Finally, in contrast to some of the existing literature on model selection, the analysis in the paper is nonasymptotic in nature, it does not require knowledge of the true model order, it is applicable to generic (random or deterministic) design matrices, and it neither requires submatrices of the design matrix to have full rank, nor does it assume a statistical prior on the values of the nonzero entries of the data vector. △ Less

Submitted 29 April, 2010; v1 submitted 15 November, 2009; originally announced November 2009.

Comments: 5 pages; Accepted for Proc. 2010 IEEE International Symposium on Information Theory (ISIT 2010)

Journal ref: Proc. IEEE Int. Symp. Information Theory, Austin, TX, Jun. 13-18, 2010, pp. 1568-1572

Performance Bounds for Expander-based Compressed Sensing in the presence of Poisson Noise

Authors: Sina Jafarpour, Rebecca Willett, Maxim Raginsky, Robert Calderbank

Abstract: This paper provides performance bounds for compressed sensing in the presence of Poisson noise using expander graphs. The Poisson noise model is appropriate for a variety of applications, including low-light imaging and digital streaming, where the signal-independent and/or bounded noise models used in the compressed sensing literature are no longer applicable. In this paper, we develop a novel… ▽ More This paper provides performance bounds for compressed sensing in the presence of Poisson noise using expander graphs. The Poisson noise model is appropriate for a variety of applications, including low-light imaging and digital streaming, where the signal-independent and/or bounded noise models used in the compressed sensing literature are no longer applicable. In this paper, we develop a novel sensing paradigm based on expander graphs and propose a MAP algorithm for recovering sparse or compressible signals from Poisson observations. The geometry of the expander graphs and the positivity of the corresponding sensing matrices play a crucial role in establishing the bounds on the signal reconstruction error of the proposed algorithm. The geometry of the expander graphs makes them provably superior to random dense sensing matrices, such as Gaussian or partial Fourier ensembles, for the Poisson noise model. We support our results with experimental demonstrations. △ Less

Submitted 25 November, 2009; v1 submitted 6 November, 2009; originally announced November 2009.

Comments: Received the best student paper award in Asilomar 2009

Construction of a Large Class of Deterministic Sensing Matrices that Satisfy a Statistical Isometry Property

Authors: Robert Calderbank, Stephen Howard, Sina Jafarpour

Abstract: Compressed Sensing aims to capture attributes of $k$-sparse signals using very few measurements. In the standard Compressed Sensing paradigm, the $\m\times \n$ measurement matrix $\A$ is required to act as a near isometry on the set of all $k$-sparse signals (Restricted Isometry Property or RIP). Although it is known that certain probabilistic processes generate $\m \times \n$ matrices that sati… ▽ More Compressed Sensing aims to capture attributes of $k$-sparse signals using very few measurements. In the standard Compressed Sensing paradigm, the $\m\times \n$ measurement matrix $\A$ is required to act as a near isometry on the set of all $k$-sparse signals (Restricted Isometry Property or RIP). Although it is known that certain probabilistic processes generate $\m \times \n$ matrices that satisfy RIP with high probability, there is no practical algorithm for verifying whether a given sensing matrix $\A$ has this property, crucial for the feasibility of the standard recovery algorithms. In contrast this paper provides simple criteria that guarantee that a deterministic sensing matrix satisfying these criteria acts as a near isometry on an overwhelming majority of $k$-sparse signals; in particular, most such signals have a unique representation in the measurement domain. Probability still plays a critical role, but it enters the signal model rather than the construction of the sensing matrix. We require the columns of the sensing matrix to form a group under pointwise multiplication. The construction allows recovery methods for which the expected performance is sub-linear in $\n$, and only quadratic in $\m$; the focus on expected performance is more typical of mainstream signal processing than the worst-case analysis that prevails in standard Compressed Sensing. Our framework encompasses many families of deterministic sensing matrices, including those formed from discrete chirps, Delsarte-Goethals codes, and extended BCH codes. △ Less

Submitted 10 October, 2009; originally announced October 2009.

Comments: 16 Pages, 2 figures, to appear in IEEE Journal of Selected Topics in Signal Processing, the special issue on Compressed Sensing

Efficient and Robust Compressed Sensing using High-Quality Expander Graphs

Authors: Sina Jafarpour, Weiyu Xu, Babak Hassibi, Robert Calderbank

Abstract: Expander graphs have been recently proposed to construct efficient compressed sensing algorithms. In particular, it has been shown that any $n$-dimensional vector that is $k$-sparse (with $k\ll n$) can be fully recovered using $O(k\log\frac{n}{k})$ measurements and only $O(k\log n)$ simple recovery iterations. In this paper we improve upon this result by considering expander graphs with expansio… ▽ More Expander graphs have been recently proposed to construct efficient compressed sensing algorithms. In particular, it has been shown that any $n$-dimensional vector that is $k$-sparse (with $k\ll n$) can be fully recovered using $O(k\log\frac{n}{k})$ measurements and only $O(k\log n)$ simple recovery iterations. In this paper we improve upon this result by considering expander graphs with expansion coefficient beyond 3/4 and show that, with the same number of measurements, only $O(k)$ recovery iterations are required, which is a significant improvement when $n$ is large. In fact, full recovery can be accomplished by at most $2k$ very simple iterations. The number of iterations can be made arbitrarily close to $k$, and the recovery algorithm can be implemented very efficiently using a simple binary search tree. We also show that by tolerating a small penalty on the number of measurements, and not on the number of recovery iterations, one can use the efficient construction of a family of expander graphs to come up with explicit measurement matrices for this method. We compare our result with other recently developed expander-graph-based methods and argue that it compares favorably both in terms of the number of required measurements and in terms of the recovery time complexity. Finally we will show how our analysis extends to give a robust algorithm that finds the position and sign of the $k$ significant elements of an almost $k$-sparse signal and then, using very simple optimization techniques, finds in sublinear time a $k$-sparse signal which approximates the original signal with very high precision. △ Less

Submitted 23 June, 2008; originally announced June 2008.

A Sublinear Algorithm for Sparse Reconstruction with l2/l2 Recovery Guarantees

Authors: Robert Calderbank, Stephen Howard, Sina Jafarpour

Abstract: Compressed Sensing aims to capture attributes of a sparse signal using very few measurements. Candès and Tao showed that sparse reconstruction is possible if the sensing matrix acts as a near isometry on all $\boldsymbol{k}$-sparse signals. This property holds with overwhelming probability if the entries of the matrix are generated by an iid Gaussian or Bernoulli process. There has been signific… ▽ More Compressed Sensing aims to capture attributes of a sparse signal using very few measurements. Candès and Tao showed that sparse reconstruction is possible if the sensing matrix acts as a near isometry on all $\boldsymbol{k}$-sparse signals. This property holds with overwhelming probability if the entries of the matrix are generated by an iid Gaussian or Bernoulli process. There has been significant recent interest in an alternative signal processing framework; exploiting deterministic sensing matrices that with overwhelming probability act as a near isometry on $\boldsymbol{k}$-sparse vectors with uniformly random support, a geometric condition that is called the Statistical Restricted Isometry Property or StRIP. This paper considers a family of deterministic sensing matrices satisfying the StRIP that are based on \srm codes (binary chirps) and a $\boldsymbol{k}$-sparse reconstruction algorithm with sublinear complexity. In the presence of stochastic noise in the data domain, this paper derives bounds on the $\boldsymbol{\ell_2}$ accuracy of approximation in terms of the $\boldsymbol{\ell_2}$ norm of the measurement noise and the accuracy of the best $\boldsymbol{k}$-sparse approximation, also measured in the $\boldsymbol{\ell_2}$ norm. This type of $\boldsymbol{\ell_2 /\ell_2}$ bound is tighter than the standard $\boldsymbol{\ell_2 /\ell_1}$ or $\boldsymbol{\ell_1/ \ell_1}$ bounds. △ Less

Submitted 17 October, 2009; v1 submitted 23 June, 2008; originally announced June 2008.