¹¹institutetext: The Hebrew University of Jerusalem ²²institutetext: Airbus Central Research & Technology, AI Research
²²email: {yizhak.elboher, raya.elsaleh, omri.isac, guy.katz}@mail.huji.ac.il
{melanie.ducoffe, audrey.galametz, guillaume.poveda, ryma.boumazouza,
noemie-lucie-virginie.cohen}@airbus.com

Robustness Assessment of a Runway Object Classifier for Safe Aircraft Taxiing

Yizhak Elboher Authors contributed equally.11 Raya Elsaleh^⋆ 11 Omri Isac^⋆ 11 Mélanie Ducoffe 22 Audrey Galametz 22 Guillaume Povéda 22 Ryma Boumazouza 22 Noémie Cohen 22 Guy Katz 11

Abstract

As deep neural networks (DNNs) are becoming the prominent solution for many computational problems, the aviation industry seeks to explore their potential in alleviating pilot workload and improving operational safety. However, the use of DNNs in these types of safety-critical applications requires a thorough certification process. This need could be partially addressed through formal verification, which provides rigorous assurances — e.g., by proving the absence of certain mispredictions. In this case-study paper, we demonstrate this process on an image-classifier DNN currently under development at Airbus, which is intended for use during the aircraft taxiing phase. We use formal methods to assess this DNN’s robustness to three common image perturbation types: noise, brightness and contrast, and some of their combinations. This process entails multiple invocations of the underlying verifier, which might be computationally expensive; and we therefore propose a method that leverages the monotonicity of these robustness properties, as well as the results of past verification queries, in order to reduce the overall number of verification queries required by nearly 60%. Our results indicate the level of robustness achieved by the DNN classifier under study, and indicate that it is considerably more vulnerable to noise than to brightness or contrast perturbations.

1 Introduction

In recent years, deep neural networks (DNNs) have been revolutionizing computer science, advancing the state of the art in many domains [16] — including natural language processing, computer vision, and many others. In the aviation domain, aircraft manufacturers are now exploring how deep-learning-based technologies could decrease the cognitive load on pilots, while increasing the safety and operational efficiency of, e.g., airports. In particular, these technologies could prove useful during the aircraft taxi phase, which often creates an increased cognitive load on pilots who have to simultaneously manage the flight plan, the aircraft itself, and any objects on the tarmac.

Despite their success, DNNs are known to be prone to various errors. Notable among these are adversarial inputs [5], which are slightly perturbed inputs that lead to incorrect and potentially unsafe DNN outputs. While there exist many techniques for efficiently finding adversarial inputs, it is unclear how to certify that no such examples exist. However, such a certification process will be required to allow the integration of DNNs into safety-critical industrial systems, e.g., in aviation.

Aviation authorities involved in managing aircraft certification, such as the European Union Aviation Safety Agency (EASA), have recently published the key elements required for certifying DNN models to be used in aviation.¹¹1https://www.easa.europa.eu/en/downloads/137631/en There, EASA particularly emphasizes that DNN verification solutions, to be applied during the learning and system integration phases, will likely constitute a means of compliance with regulatory requirements.²²2https://www.easa.europa.eu/en/document-library/general-publications/concepts-design-assurance-neural-networks-codann EASA points out, however, that the current scalability and the expressiveness of DNN verification techniques is limited.

Typically, DNN formal verification tools seek to prove that, for a given infinite set of inputs, a DNN only produces outputs that fall within a safe subspace of the output space. To date, these tools have been predominantly applied in assessing the robustness of DNN predictions against specific types of local input perturbations. Maturing these techniques is thus key in allowing them to meet the bar needed for DNN certification in, e.g., aviation. This point is again stressed in EASA’s AI Roadmap,³³3https://www.easa.europa.eu/en/domains/research-innovation/ai which emphasizes the need for providing more general guarantees of a DNN’s stability.

Although DNN verification has been making great strides [1, 7, 6, 12, 18, 14, 15, 8, 13, 11], it has so far been applied only to a limited number of real-world systems. In this case-study paper, we study the applicability and scalability of DNN verification through an object classification use-case, relevant to the aviation domain and of specific interest to Airbus. We explore pertinent vision-oriented perturbations (noise, brightness, and contrast) and use formal verification to quantify their effects on DNN’s robustness. As a back-end engine, we use the Marabou DNN verifier [17]. We also demonstrate that the verification process can be optimized by leveraging the monotonicity of the studied perturbations.

Our results indicate that while the DNN is highly sensitive to noise perturbations, it is slightly less vulnerable to contrast and brightness perturbations. This is a reassuring result, as these perturbations are strongly correlated with highly unpredictable operating conditions, especially outdoors. More broadly, our results showcase the usefulness and potential of DNN verification in aviation that could easily be extended to other safety-critical domains.

2 Background

Deep Neural Networks. A deep neural network [4] $\mathcal{N}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{k}$ is comprised of $m$ layers, $L_{1},...,L_{m}$ . Each layer $L_{i}$ consists of a set of nodes, $S_{i}$ . When $\mathcal{N}$ is evaluated, each node in the input layer is assigned an initial value. Then, the value of the $j^{th}$ node in the $2\leq i<m$ layer, $v_{j}^{i}$ , is computed as:

$v_{j}^{i}=f\left(\underset{l=1}{\overset{|S_{i-1}|}{\sum}}w^{i-1}_{j,l}\cdot v% ^{i-1}_{l}+b^{i}_{j}\right)$

where $f:\mathbb{R}\rightarrow\mathbb{R}$ is an activation function and $w^{i-1}_{j,l},b^{i}_{j}\in\mathbb{R}$ are the respective weights and biases of $\mathcal{N}$ . The most common activation function is the rectified linear unit (ReLU), defined as $\text{ReLU}{}(x)=max(0,x)$ . Finally, neurons in the output layer are assigned values using an affine combination only. The output of the DNN is the values of the nodes in its final layer. An image-classifier $\mathcal{N}:\mathbb{R}^{n}\rightarrow C\subset\mathbb{N}$ assigns each input image $x^{\prime}$ a class $c\in C$ , which describes the main object depicted in $x^{\prime}$ . For convenience, $x^{\prime}$ is regarded as both a vector and a matrix, interchangeably. For an example of a DNN and its evaluation, see Appendix B.

DNN Verification. For a DNN $\mathcal{N}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{k}$ , input property $P\subset\mathbb{R}^{n}$ and output property $Q\subset\mathbb{R}^{k}$ , the DNN verification problem is to decide whether there exist $x\in P$ and $y\in Q$ such that $\mathcal{N}(x)=y$ . If such a pair exists, the verification query $(\mathcal{N},P,Q)$ is satisfiable (SAT), and the pair $(x,y)$ is called a witness; otherwise, it is unsatisfiable (UNSAT). Typically, $Q$ encodes an undesired behavior, and so a witness is a counterexample that demonstrates an error.

3 Industrial Use-Case: Runway Object Classification

3.1 Runway Object Classification

In 2020, Airbus concluded its Autonomous Taxi, Take-Off and Landing (ATTOL) project.⁴⁴4https://www.airbus.com/en/newsroom/press-releases/2020-06-airbus-concludes-attol-with-fully-autonomous-flight-tests The objective of ATTOL was to design a fully autonomous controller for the taxi, take-off, approach and landing phases of a commercial aircraft — by leveraging state-of-the-art technology, and in particular deep-learning models used for vision-assisted functions. As part of the project, $400$ flights over a period of two years were instrumented to collect video data from aircraft in operation. This unique dataset is currently being used to further mature several vision-based functions within Airbus. Using this dataset, it was observed that the taxi phase of the flight, in particular, could benefit from autonomous support. During this phase, pilots are conducting aircraft operations, while simultaneously dealing with the unpredictable nature of airport management and traffic. Object identification, in particular of potential threats on the runway, could thus support the pilots during this phase. Several object classifiers are being tested for this purpose within Airbus.

In this study, we focus on images of runway objects extracted from taxiing videos — i.e., all objects are observed from an aircraft on the ground. We extract $(224\times 224)$ pixel images from the original, high-resolution gray-scaled images, centered on a specific runway object. A DNN $N_{1}$ is trained on resampled $(32\times 32)$ images. The four considered classes are Aircraft, Vehicle, Person, and Negative, extracted where no object is found. $N_{1}$ is a feedforward DNN, with roughly $8000$ ReLU neurons, and an accuracy of $85.3\%$ on the test dataset ( $1145/1342$ images).⁵⁵5These DNNs will not be used as such in Airbus products. More robust models are currently under development, in part supported by analyses such as the one presented here.

3.2 Properties of Interest

We seek to verify the local robustness of a runway object classifier $\mathcal{N}$ ; i.e., that small perturbations around a correctly-classified input $x^{\prime}$ do not cause misclassification, encoded by $Q$ . We specify $Q$ as: $Q_{x^{\prime}}:=C\setminus\mathcal{N}(x^{\prime})$ . We use the input property $P$ to define three perturbation types: noise, brightness, and contrast.

Noise. In this widely studied form of perturbation [10, 2], the perturbed input images are taken from an $\epsilon$ -ball around $x^{\prime}$ : $P=B_{\epsilon}(x^{\prime})$ , where $B_{\epsilon}$ is the $\ell_{\infty}$ - $\epsilon$ -ball around $x^{\prime}$ , and $\epsilon>0$ .

Brightness. A brightness perturbation is caused by shifting all pixels of $x^{\prime}$ by a constant value $b$ : $bright(x^{\prime},b):=x^{\prime}+b\cdot J_{n}$ , where $J_{n}$ is the all-ones matrix of size $n\times n$ . We define $P=bright_{\beta}(x^{\prime}):=\{bright(x^{\prime},b)\mid|b|\leq\beta\}$ for some $\beta>0$ , to allow all brightness perturbations of absolute value at most $\beta$ . See Appendix A for a visual example.

Contrast. A contrast perturbation $con(x^{\prime},c,\mu)$ is created by scaling all image pixels multiplicatively, rescaling their difference from a mean value $\mu\in[0,1]$ by a multiplicative constant $c\in\mathbb{R}_{\geq 0}$ : $con(x^{\prime},c,\mu):=\mu\cdot J_{n}+c\cdot(x^{\prime}-\mu\cdot J_{n})$ . We then set $P=con_{\gamma,\mu}(x^{\prime}):=\{con(x^{\prime},c,\mu)\mid|c-1|\leq\gamma\}$ , to encode all contrast perturbations with value of at most $\gamma$ , where $\mu$ remains constant and $\gamma\in[0,1]$ . See Appendix A for a visual example.

4 The Formal Verification Process

4.1 Encoding Brightness and Contrast Perturbations

We now show how to encode the brightness and contrast properties described in Section 2 into verification queries that assess robustness to noise perturbations over a modified input space. This reduction allows us to use any of the available tools that support such queries as a backend. The encoding is performed by adding a new input layer to the network, as illustrated in Fig. 1.

(a) Brightness. Weights are set to

1

, biases to

0

(b) Contrast. Biases are set to

\mu

Figure 1: Modeling brightness and contrast perturbations by adding an input layer.

Brightness. The new input layer clones the original input layer and adds a single neuron $b$ to represent the brightness perturbations. The weights from the new layer to the following, original input layer are set to $1$ so that every variable $x_{i}\in x$ is assigned $x_{i}=x^{\prime}_{i}+b$ . The bounds for the new neuron are set to $b\leq\beta,b\geq-\beta$ , whereas inputs $x_{i}^{\prime}$ are exactly restricted to the input around which the robustness is being verified. We note that in this case, this single construction allows the verification of the robustness around any input, by selecting appropriate $x_{i}^{\prime}$ values. We further note that this construction can be used to simultaneously encode noise and brightness perturbations, by bounding the input neurons $x_{i}^{\prime}$ to an $\epsilon$ -ball around an input of interest. This gives rise to two-dimensional queries, for any combination of $\beta$ and $\epsilon$ values, which allows modeling a more realistic nature of perturbations.

Contrast. The new input layer contains a single input neuron, $c$ . We treat $\mu,x^{\prime}$ as constants, and set the weights from the new layer in a way that every neuron $x_{i}$ in that layer is assigned $x_{i}=(x^{\prime}_{i}-\mu)\cdot c+\mu$ . Finally, we set the bounds $c\geq 1-\gamma,c\leq 1+\gamma$ . We note that the contrast perturbation is multiplicative with respect to $c$ , $\mu$ , and the input image $x^{\prime}$ . Since DNN verification algorithms typically only support linear operations, either $x^{\prime}$ or $c$ should be fixed. Therefore, a separate DNN is constructed for each input image; and there is no immediate way to encode a simultaneous noise perturbation.

4.2 Incremental Verification Algorithm

For any fixed image $x^{\prime}$ , we seek to solve numerous brightness, noise and contrast robustness queries, with different values of $\epsilon$ , $\beta$ and $\gamma$ . Since executing these queries is computationally expensive, we exploit the monotonicity of these properties to reduce their number. Let $\beta^{\prime}<\beta$ , $\epsilon^{\prime}<\epsilon$ and $\gamma^{\prime}<\gamma$ . If there exists an adversarial example for parameters $\beta^{\prime}$ , $\epsilon^{\prime}$ or $\gamma^{\prime}$ , it then also constitutes a counterexample for a query with parameters $\beta$ , $\epsilon$ or $\gamma$ respectively. Conversely, if the network is robust with respect to parameters $\beta$ , $\epsilon$ or $\gamma$ , then it is also robust to perturbation with parameters $\beta^{\prime},\epsilon^{\prime}$ or $\gamma^{\prime}$ respectively.

We exploit this property in a binary search algorithm for contrast queries, and in our incremental verification algorithm for brightness and noise queries. Intuitively, the algorithm initializes a grid representing all combinations of $\epsilon,\beta$ parameters that need to be verified. The observation above states that for every row and every column, there is at most one transition from UNSAT to SAT, which is represented by a step graph within the grid. The algorithm then discovers this step graph instead of solving all queries in the grid.

The pseudo-code of the algorithm appears in Algorithm 1. Formally, Algorithm 1 assumes the existence of a verification procedure $\text{verify}(\mathcal{N},x^{\prime},\beta,\epsilon)$ which verifies the robustness of a network $\mathcal{N}$ to noise perturbations of value at most $\epsilon$ and brightness perturbations of value at most $\beta$ around an image $x^{\prime}$ . The algorithm is given an input network $\mathcal{N}$ , an image $x^{\prime}$ , and two increasingly ordered arrays $B,E$ , containing the values of parameters $\beta$ and $\epsilon$ we intend to check, respectively. Then, the algorithm initializes a grid representing all possible combinations of parameters $(\beta,\epsilon)\in B\times E$ , and a temporary tuple $(b,e)$ representing the lowest value of $\beta$ and highest value $\epsilon$ and corresponding to the top-left corner of the grid. The algorithm then iteratively calls $\text{verify}(\mathcal{N},x^{\prime},b,e)$ to populate the grid. If the result is SAT, then for all queries with the same $\epsilon$ value, and a greater $\beta$ (all cells to the right of the current cell) the result is SAT as well.⁶⁶6Note that this is the case for all queries with greater values of $\epsilon,\beta$ (the top right rectangle), though the values of queries with a greater $\epsilon$ value are already decided. A dual argument applies to the UNSAT case as well. We then mark the relevant cells with SAT and decrement $\epsilon$ to the next value. If the result is UNSAT, then for all queries with the same $\beta$ value, and a smaller $\epsilon$ (all cells to the bottom of the current cell), the result is UNSAT as well. We then mark the relevant cells with UNSAT and increment $\beta$ to the next value. When the current cell reaches the final column or row, a typical binary search algorithm is used to find the remaining results. Note that the algorithm requires only $O(m)$ calls to the verifier while naively solving all queries requires $O(m^{2})$ calls, where $m$ is the number of possible values of $\beta$ or $\epsilon$ (the maximal of the two). For contrast queries, the binary search allows using a logarithmic number of invocations of the verifier instead of a linear number.

To support the use of real-world verifiers, we also address cases where the verifier returns a TIMEOUT or error value. When these cases occur, we mark the corresponding cell $(e,b)$ with an UNKNOWN result, and increment the value of $b$ as if the result was SAT. In addition, we use binary search for the remaining values of $e$ , where the value of $b$ is constant. Note that in the presence of TIMEOUT, the bound of $O(m)$ calls to the verifier is not guaranteed.

Input: Arrays $B,E$ with values of $\epsilon,\beta$ in increasing order, respectively, a verifier V, a network $\mathcal{N}$ and an image $x^{\prime}$ .
Output: A grid representing the robustness of $\mathcal{N}$ to brightness and noise perturbations around $x^{\prime}$ , for all values in $B,E$ .

b\leftarrow 0

e\leftarrow length(E)-1

grid\leftarrow 0_{length(B)\times length(E)}

while

b<length(B)

and

e\geq 0

b=length(B)-1

then

Binary search with remaining values of

e

;

b

is constant.

end if

e=0

then

Binary search with remaining values of

b

;

e

is constant.

end if

result\leftarrow V.\text{verify}(\mathcal{N},x^{\prime},E[e],B[b])

result=\texttt{SAT}

then

\forall i\geq b:grid[i][e]\leftarrow\texttt{SAT}

e\leftarrow e-1

else if

result=\texttt{UNSAT}

then

\forall j\leq e:grid[b][j]\leftarrow\texttt{UNSAT}

b\leftarrow b+1

else

\triangleright

Timeout, Memoryout, etc.

grid[b][e]\leftarrow\texttt{UNKNOWN}

Binary search with remaining values of

e

;

b

is constant.

b\leftarrow b+1

end if

end while

return grid

Algorithm 1 Incremental verification algorithm

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 2: Example of incremental verification algorithm’s run.

Example. In Fig. 2, the grid represents the options for verification queries of robustness for brightness and noise perturbation, with parameters $(\beta,\epsilon)\in[0.1,0.2,0.3,0.4,0.5]\times[0.1,0.2,0.3,0.4]$ . The purple cell represents the current tuple $(\beta,\epsilon)$ . Red marks UNSAT queries, green marks SAT queries. Rich colors represent a call for the verifier, while pale colors represent a deduction of satisfiability. The algorithm first queries the verifier to verify robustness with parameters $(0.4,0.1)$ , which returns UNSAT. Then, the algorithm deduces UNSAT for queries with $\beta=0.1,\epsilon<0.4$ without calling the verifier again, and queries the verifier to verify robustness with parameters $(0.4,0.2)$ . Since the verifier returns UNSAT again, the algorithm deduces UNSAT for queries with $\beta=0.2,\epsilon<0.4$ and queries the verifier to verify robustness with parameters $(0.4,0.3)$ . This time, the verifier returns SAT, so the algorithm deduces SAT for queries with $\beta>0.2,\epsilon=0.4$ . The algorithm then queries the verifier to verify robustness with parameters $(0.3,0.3)$ . The rest of the iterations continue similarly.

5 Evaluation

For the $1145$ correctly classified test images, we verify $N_{1}$ ’s robustness to noise and brightness for parameters $(\epsilon,\beta)\in[0,0.05,0.1,0.15,0.2]\times[0,0.1,0.2,0.3,0.4,0.5]$ , and to contrast perturbations with mean pixel value $\mu=0.2585$ and $\gamma\in[0.1,0.2,...,0.9]$ . We make use of the incremental verification algorithm for noise and brightness perturbations. For contrast, we run a binary search algorithm to find the minimal $\gamma$ parameter for which the query is UNSAT. We use an arbitrary timeout of $22.5K$ seconds per single query, and $80$ hours for the overall runtime to analyze a single input point. The results are summarized below and in Appendix C.

Refer to caption — Figure 3: Percentage of UNSAT queries per noise and brightness parameters.

Fig. 3 shows the percentage of UNSAT queries for noise and brightness perturbations, indicating the absence of counter-examples, of 1097 points for which the analysis has not timed out. The incremental verification algorithm invoked the verifier on 13231 queries, whereas the results of 59% of the queries were deduced, using the incremental approach, without additional invocations. Fig. 4 shows the percentage of UNSAT queries for contrast perturbations within the range $[1-\gamma,1+\gamma]$ . The binary search algorithm invoked the verifier 3915 times, whereas the remaining 62% of the queries were deduced without additional invocations. For contrast perturbations, all queries terminated without a timeout.

Overall, the results indicate that the classifier shows similar robustness to contrast and brightness perturbations. However, it is significantly more sensitive to noise perturbations. We note that noise in images comes from various sources. Some noise is inherent to the camera’s sensor (e.g., impulse noise, thermal noise) or its associated electronics (e.g. shot noise). Other noise is a direct consequence of operating and environmental conditions (e.g., low-light conditions, scenery colors, etc.). Brightness and contrast also fall into this category; they are both inherently related to operating conditions. Although noise originating from image acquisition is certainly a nuisance, it can in part be reduced by noise reduction techniques, as well as an expert understanding of the camera characteristics and continuous quality tracking. Other kinds of noise are more challenging to predict or mitigate, as the number of different operating conditions (weather, time of day, scenery, etc.) is effectively infinite. Therefore, it is somehow reassuring that our classifier seems to be less vulnerable to contrast and brightness, as these perturbations are highly unpredictable.

6 Conclusion

As numerous state-of-the-art image classifiers are vulnerable to small image perturbations, robustness is a key safety requirement; and certification authorities, such as EASA, might require confirmed robustness as part of the model certification process in the aerospace domain.⁷⁷7https://www.easa.europa.eu/en/easa-concept-paper-first-usable-guidance-level-1- machine-learning-applications-proposed-issue-01pdf This work explores the challenges that the industry is facing in its effort to safely deploy deep-learning-based systems, and the benefits that formal methods can afford in assessing the robustness of DNN models to various perturbations. One significant challenge is the limited scalability of current verification techniques.

In this work we focused on assessing the robustness of a prototype runway object classifier provided by Airbus, with respect to three common image perturbations types. To partially address the scalability challenge, we exploited the monotonicity of these perturbations in designing an algorithm that improved the performance of the overall verification process. Moving forward, we aim to assess additional, larger, Airbus networks with higher-resolution input; and to verify their robustness to simultaneous brightness and contrast perturbations. To improve performance, which will enable verifying larger networks, we intend to examine applying DNN abstraction methods [3] to the verification queries we have used. In addition, we aspire to increase the reliability of the results by using the proof producing version of Marabou [9].

Acknowledgements. This research was partially funded by Airbus Central Research & Technology, AI Research and by the Israeli Smart Transportation Research Center (ISTRC).

References

Brix et al. [2023] Brix, C., Müller, M., Bak, S., Johnson, T., Liu, C.: First Three Years of the International Verification of Neural Networks Competition (VNN-COMP). Int. Journal on Software Tools for Technology Transfer pp. 1–11 (2023)
Casadio et al. [2022] Casadio, M., Komendantskaya, E., Daggitt, M., Kokke, W., Katz, G., Amir, G., Refaeli, I.: Neural Network Robustness as a Verification Property: A Principled Case Study. In: Proc. 34th Int. Conf. on Computer Aided Verification (CAV). pp. 219–231 (2022)
Elboher et al. [2020] Elboher, Y., Gottschlich, J., Katz, G.: An Abstraction-Based Framework for Neural Network Verification. In: Proc. 32nd Int. Conf. on Computer Aided Verification (CAV). pp. 43–65 (2020)
Goodfellow et al. [2016] Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press (2016)
Goodfellow et al. [2014] Goodfellow, I., Shlens, J., Szegedy, C.: Explaining and Harnessing Adversarial Examples (2014), technical Report. http://arxiv.org/abs/1412.6572
Gopinath et al. [2018] Gopinath, D., Katz, G., Pǎsǎreanu, C., Barrett, C.: DeepSafe: A Data-driven Approach for Assessing Robustness of Neural Networks. In: Proc. 16th. Int. Symposium on Automated Technology for Verification and Analysis (ATVA). pp. 3–19 (2018)
Henriksen and Lomuscio [2020] Henriksen, P., Lomuscio, A.: Efficient Neural Network Verification via Adaptive Refinement and Adversarial Search. In: Proc. 24th European Conf. on Artificial Intelligence (ECAI). pp. 2513–2520 (2020)
Huang et al. [2017] Huang, X., Kwiatkowska, M., Wang, S., Wu, M.: Safety Verification of Deep Neural Networks. In: Proc. 29th Int. Conf. on Computer Aided Verification (CAV). pp. 3–29 (2017)
Isac et al. [2022] Isac, O., Barrett, C., Zhang, M., Katz, G.: Neural Network Verification with Proof Production. In: Proc. 22nd Int. Conf. on Formal Methods in Computer-Aided Design (FMCAD). pp. 38–48 (2022)
Katz et al. [2021] Katz, G., Barrett, C., Dill, D., Julian, K., Kochenderfer, M.: Reluplex: a Calculus for Reasoning about Deep Neural Networks. Formal Methods in System Design (FMSD) (2021)
Lyu et al. [2020] Lyu, Z., Ko, C.Y., Kong, Z., Wong, N., Lin, D., Daniel, L.: Fastened Crown: Tightened Neural Network Robustness Certificates. In: Proc. 34th AAAI Conf. on Artificial Intelligence (AAAI). pp. 5037–5044 (2020)
Mangal et al. [2019] Mangal, R., Nori, A., Orso, A.: Robustness of Neural Networks: A Probabilistic and Practical Approach. In: Proc. IEEE/ACM 41st Int. Conf. on Software Engineering: New Ideas and Emerging Results (ICSE-NIER). pp. 93–96 (2019)
Müller et al. [2022] Müller, M., Makarchuk, G., Singh, G., Püschel, M., Vechev, M.: PRIMA: General and Precise Neural Network Certification via Scalable Convex Hull Approximations. In: Proc. 49th ACM SIGPLAN Symposium on Principles of Programming Languages (POPL) (2022)
Ostrovsky et al. [2022] Ostrovsky, M., Barrett, C., Katz, G.: An Abstraction-Refinement Approach to Verifying Convolutional Neural Networks. In: Proc. 20th. Int. Symposium on Automated Technology for Verification and Analysis (ATVA). pp. 391–396 (2022)
Singh et al. [2019] Singh, G., Gehr, T., Puschel, M., Vechev, M.: An Abstract Domain for Certifying Neural Networks. In: Proc. 46th ACM SIGPLAN Symposium on Principles of Programming Languages (POPL) (2019)
Szegedy et al. [2013] Szegedy, C., Toshev, A., Erhan, D.: Deep Neural Networks for Object Detection. Advances in Neural Information Processing Systems 26 (2013)
Wu et al. [2024] Wu, H., Isac, O., Zeljić, A., Tagomori, T., Daggitt, M., Kokke, W., Refaeli, I., Amir, G., Julian, K., Bassan, S., Huang, P., Lahav, O., Wu, M., Zhang, M., Komendantskaya, E., Katz, G., Barrett, C.: Marabou 2.0: A Versatile Formal Analyzer of Neural Networks. In: Proc. 36th Int. Conf. on Computer Aided Verification (CAV) (2024)
Wu et al. [2022] Wu, H., Zeljić, A., Katz, G., Barrett, C.: Efficient Neural Network Analysis with Sum-of-Infeasibilities. In: Proc. 28th Int. Conf. on Tools and Algorithms for the Construction and Analysis of Systems (TACAS). pp. 143–163 (2022)

Appendix

A Visualization of Brightness and Contrast Perturbations

B An Example of DNN

Consider the DNN with $4$ layers that appears in Fig. 9, where all biases are set to zero and are ignored. For input $\langle 2,-1\rangle$ , the first node in the second layer evaluates to $\text{ReLU}{}(2\cdot 1.5\;+\;-1\cdot(-1))=\text{ReLU}{}(4)=4$ ; and the second node in the second layer evaluates to $\text{ReLU}{}(2\cdot-1)=\text{ReLU}{}(-2)=0$ ; Then the node in the third layer evaluates to $\text{ReLU}{}(4-0)=4$ . and thus the output of the network is $2$ .

Figure 9: A toy DNN.

C Detailed Evaluation Results

In this appendix, we provide a detailed description of our experiments’ results, as described in Section 5.

Result	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9
SAT	20.36	20.57	21.44	25.32	54.26	28.36	27.28	37.41	36.37
UNSAT	19.06	19.06	21.93	106.14	29.37	284.44	241.34	87.12	63.89
UNKNOWN	0	0	0	0	0	0	51.77	59.21	55.07

Table 1: Avg. solving time for contrast queries (sec.).

#		0	0.1	0.2	0.3	0.4	0.5
SAT	0	N/A	49.52	61.47	45.09	53.15	66.37
UNSAT		N/A	79.62	73.36	69.92	74.94	83.97
SAT	0.05	50.99	38.27	46.42	373.89	1457.50	1125.45
UNSAT		315.12	423.99	787.40	1111.39	652.05	759.83
SAT	0.1	51.24	48.28	34.85	39.99	422.11	112.02
UNSAT		1887.37	2027.04	159.66	72.11	89.05	301.96
SAT	0.15	39.44	33.46	1474.75	745.56	26.68	22.13
UNSAT		2359.39	225.52	84.77	393.32	259.87	1026.77
SAT	0.2	35.63	18.64	N/A	18.61	N/A	N/A
UNSAT		77.30	92.12	1165.06	1489.30	N/A	N/A

Table 2: Avg. solving time for noise and brightness queries (sec.).

# Queries	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9
SAT Overall	42	80	136	203	306	465	654	836	901
deduced	0	0	80	136	0	326	306	654	836
verified	42	80	56	67	306	159	348	182	65
UNSAT Overall	1103	1065	1009	942	839	680	490	308	241
deduced	1065	839	839	839	0	490	0	0	0
verified	38	226	170	103	839	190	490	308	241
UNKNOWN	0	0	0	0	0	0	1	1	3

Table 3: Results summary for contrast queries.

#		0	0.1	0.2	0.3	0.4	0.5
SAT Overall	0	0	24	53	84	109	139
deduced		0	0	0	53	53	109
verified		0	24	53	31	56	30
UNSAT Overall		1097	1073	1044	1013	988	958
deduced		1097	1044	235	889	67	51
verified		0	29	809	124	921	907
SAT Overall	0.05	325	371	424	484	533	1093
deduced		0	325	371	325	483	533
verified		325	46	53	159	50	560
UNSAT Overall		235	194	146	100	69	52
deduced		14	10	8	6	6	3
verified		221	184	136	94	63	49
SAT Overall	0.1	829	859	884	900	934	936
deduced		0	829	859	829	900	934
verified		829	30	25	71	34	2
UNSAT Overall		14	10	8	6	6	6
deduced		4	4	3	3	3	0
verified		10	6	5	3	3	6
SAT Overall	0.15	1036	1045	1054	1064	1070	1075
deduced		0	1036	1047	1036	1064	1070
verified		1036	11	7	28	6	5
UNSAT Overall		4	4	3	3	3	3
deduced		2	1	1	1	0	0
verified		2	3	2	2	3	3
SAT Overall	0.2	1087	1091	1091	1091	1091	1091
deduced		0	1087	1091	1087	1091	1091
verified		1087	4	0	4	0	0
UNSAT Overall		2	1	2	1	0	0
deduced		0	0	0	0	0	0
verified		2	1	2	1	0	0

Table 4: Summary of results for noise and brightness queries (excluding timeouts).

D The Network is more Robust to Brightness Perturbations than to Noise

In both Fig. 10 and Fig. 11, we show the percentage of the verification queries that had an UNSAT result with respect to certain brightness and noise perturbations. In Fig. 10, we notice that changes in the brightness perturbation parameter $\beta$ have a minor effect on the queries’ answers. On the other hand, as emphasized in Fig. 11, changes in the noise perturbation parameter dramatically decrease the network’s performance. This observation tells us that the network is more robust and handles brightness perturbations better than noise.