Treatment of Epistemic Uncertainty in Conjunction Analysis with Dempster-Shafer Theory

L. Sanchez ¹¹1PhD Candidate, Aerospace Centre of Excellence, luis.sanchez-fdez-mellado@strath.ac.uk. and M. Vasile²²2Professor at University of Strathclyde and Director of Aerospace Centre of Excellence, massimiliano.vasile@strath.ac.uk. Aerospace Centre of Excellence. University of Strathclyde, Glasgow, G1 1XJ, United Kingdom S. Sanvido³³3Space Debris Engineer. IMS Space Consultancy GmbH, silvia.sanvido@ext.esa.int. Space IMS Space Consultancy GmbH, 64297, Darmstadt, Germany K. Mertz ⁴⁴4Spaces Debris Engineer, Space Debris Office, ESA/ESOC, klaus.merz@esa.int. Space Debris Office, European Space Operations Center (ESOC), European Space Agency (ESA), Darmstadt, 64293, Germany C. Taillan⁵⁵5Space Surveillance Engineer. Space Security, Safety and Sustainability Office, CNES, christophe.taillan@cnes.fr. Space Security, Safety and Sustainability Office, Centre National d’Etudes Spatiales (CNES), Toulouse, 31401, France

Abstract

The paper presents an approach to the modelling of epistemic uncertainty in Conjunction Data Messages (CDM) and the classification of conjunction events according to the confidence in the probability of collision. The approach proposed in this paper is based on Dempster-Shafer Theory (DSt) of evidence and starts from the assumption that the observed CDMs are drawn from a family of unknown distributions. The Dvoretzky–Kiefer–Wolfowitz (DKW) inequality is used to construct robust bounds on such a family of unknown distributions starting from a time series of CDMs. A DSt structure is then derived from the probability boxes constructed with DKW inequality. The DSt structure encapsulates the uncertainty in the CDMs at every point along the time series and allows the computation of the belief and plausibility in the realisation of a given probability of collision. The methodology proposed in this paper is tested on a number of real events and compared against existing practices in the European and French Space Agencies. We will show that the classification system proposed in this paper is more conservative than the approach taken by the European Space Agency but provides an added quantification of uncertainty in the probability of collision.

Abstract

Keywords: Space Traffic Management, Conjunction Data Message, Epistemic Uncertainty, Dempster-Shafer theory of evidence, Conjunction Assessment, Decision-making.

AI: Artificial Intelligence
Bel: Belief
bpa: basic probability assignment
CAM: Collision Avoidance Manoeuvre
CARA: Conjunction Assessment Risk Analysis
CDF: Cumulative Distribution Function
CDM: Conjunction Data Message
CNES: Centre National d’Etudes Spatiales
DKW: Dvoretzky–Kiefer–Wolfowitz
DoU: Degree of Uncertainty
DSt: Dempster-Shafer theory of evidence
eCDF: empirical Cumulative Distribution Function
ESA: European Space Agency
ESOC: European Space Operations Centre
FE: Focal Element
FN: False Negative
FP: False Positive
FPR: False Positive Rate
HBR: Hard-Body Radius
IDSS: Intelligent Decision Support System
JAC: Java for Assessment of Conjunctions
KS: Kolmogorov-Smirnov
LEO: Low Earth Orbit
ML: Machine Learning
mWSM: modified Weighted Sum Method
Pl: Plausibility
PoC: Probability of Collision
ROC: Receiver Operating Characteristic
SEM: Space Environment Management
SDO: Space Debris Office
sPoC: scaled Probability of Collision
STM: Space Traffic Management
TCA: Time of Closest Approach
TN: True Negative
TOPSIS: Technique for Order of Preference by Similarity to Ideal Solution
TP: True Positive
TPR: True Positive Rate
WPM: Weighted Product Method
WSM: Weighted Sum Method

1 Introduction

The close encounter of two space objects, also known as a conjunction between a chaser and a target, can lead to a collision if the relative position of the two objects is not properly controlled. The Probability of Collision (PoC) to happen depends on the probability that each of the two objects occupies a given position in space. This probability can be derived from the knowledge of the orbit of the two objects and the associated uncertainty.

It is customary to assume that the distribution of possible positions of the two objects at the time of closest encounter follows a multivariate Gaussian with a given mean and covariance matrix, see [1, 2]. This assumption is limited by three sources of uncertainty: the uncertainty in the dynamic model used to propagate the orbit from the last available observation to the time of closest approach, the uncertainty in the actual distribution at the time of closest approach, and the uncertainty in the last observed state before closest approach. We argue that all three forms of uncertainty are epistemic in nature since they derive from a lack of knowledge of the model, distribution and error in the observations.

The information on a given close encounter is generally available in the form of a Conjunction Data Message (CDM), which contains the means and covariances of the two objects at Time of Closest Approach (TCA), see [3]. Thus, in this paper, we start from the assumption that the mean and covariance in each CDM are affected by epistemic uncertainty, which is reflected in an uncertainty in the correct value of the PoC.

The general attempt to compensate for the uncertainty in the CDMs is to improve the realism of the covariance matrix by improving its propagation, [4], or by some form of updating of the dynamic model, [5]. These approaches are all very valuable but require direct access to the post-observation data. Other methods based solely on the available CDMs tried to predict the next CDMs using machine learning starting from an available time series, see [6, 7, 8, 9], or increased the last covariance under the assumption that the series of CDMs should follow a given distribution, [10, 11]. This last approach does not modify the mean value or miss distance.

So far, only a limited number of authors have directly addressed epistemic uncertainty in conjunction analysis, see for example [12, 13, 14, 15]. In [16, 17, 18] the authors proposed a robust approach to conjunction analysis and collision avoidance planning based on Dempster-Shafer theory of evidence (DSt). DSt allows making decisions informed by the degree of confidence in the correctness of a value rather then by the value itself, [19]. However, the available information to build the frame of discernment that is needed in DSt is often limited in a sequence of CDMs. CDMs contain little information on the three forms of uncertainty listed above and essentially only provide covariance and mean value of the miss distance. Thus, one key question is how to translate the time series of CDMs into the frame of discernment used in DSt. The underlying assumption in this work is that the CDMs are observables drawn from an unknown family of distributions defined within some bounds. Without uncertainty, one would be able to exactly predict the next CDMs as the mean and covariance would only depend on observations with a known distribution and there would be no uncertainty in the propagation model and distribution at TCA. Furthermore, we assumed that the CDMs computed from observations acquired close to the TCA were less affected by model and distribution uncertainty. This is reasonable as the propagation time is shorter and thus both nonlinearities and model errors have a lower impact on the propagation of the distribution of the possible states.

The paper introduces a methodology, based on the Dvoretzky–Kiefer–Wolfowitz (DKW) inequality, [20], to derive a DSt structure capturing the epistemic uncertainty in a given sequence of CDMs. From the DSt structures, one can compute the Belief (Bel) and Plausibility (Pl) that the value of the PoC is correct and an upper and lower bound on its value. The paper then proposes a classification system that exploits the use of Bel and Pl to differentiate between events that are uncertain from events that can lead to a collision. The overall methodology is tested on a number of real conjunction scenarios with known sequences of CDMs and compared against current practices in the European Space Agency (ESA) and Centre National d’Etudes Spatiales (CNES).

The rest of the paper is structured as follows. Section 2 briefly introduces a methodology previously presented by the authors to deal with epistemic uncertainty for risk assessment in space encounters. Section 3 extends this methodology to deal with sequence of CDMs. In Section 4, some numerical cases are presented showing the operation of the proposed method and comparing the approach with the procedure followed by real operators. Finally, Section 5 concludes the paper with the final remarks and future work.

2 Conjunction Analysis with Dempster-Shafer Structures

This section briefly introduces the basic idea of DSt applied to Conjunction Assessment Risk Analysis (CARA). It also includes the DSt-based conjunction classification system already introduced by the authors in previous works. More details on DSt can be found in [21], and more details on its application to space conjunction assessment can be found in [16, 17, 18].

In this paper, we consider only fast encounters between two objects: object 1 and object 2. Under the typical modelling assumptions of fast encounters, see [22], the PoC can be defined as:

PoC=\frac{1}{2\pi\sqrt{\|\mathbf{\Sigma}\|}}\int\limits_{\mathcal{B}((0,0),R)}% e^{-\frac{1}{2}\left((\mathbf{b}-\bm{\mu})^{T}\mathbf{\Sigma}^{-1}(\mathbf{b}-% \bm{\mu})\right)}d\xi d\zeta

(1)

where, without loss of generality, object 2 is at the centre of the coordinate system of the impact plane at the time of closest approach (TCA), $\mathbf{b}=[\xi,\zeta]^{T}$ is the position vector of object 1 with respect to object 2 projected onto the impact plane, $\bm{\Sigma}$ is the $2\times 2$ combined covariance matrix of the position of the two objects in the impact plane ( $\bm{\Sigma}=\bm{\Sigma}_{1}+\bm{\Sigma}_{2}$ , with $\bm{\Sigma}_{1}$ and $\bm{\Sigma}_{2}$ the individual covariance matrices of object 1 and 2 respectively) and $\bm{\mu}=[\mu_{\xi},\mu_{\zeta}]^{T}$ is the expected position vector of object 1 with respect to object 2 projected onto the impact plane. In the remainder of the paper $\bm{\mu}$ is called miss distance. The integration region ${\mathcal{B}((0,0),R)}$ , or Hard-Body Radius (HBR), is a disk with radius $R$ centred at the origin of the impact plane.

When the covariance $\bm{\Sigma}$ and miss distance $\bm{\mu}$ are not precisely known the $PoC$ is affected by a degree of uncertainty. This lack of knowledge translates into an epistemic uncertainty in the exact value of $\bm{\Sigma}$ and $\bm{\mu}$ . The epistemic uncertainty in covariance $\bm{\Sigma}$ and miss distance $\bm{\mu}$ can come from incertitude in the sources of information, from poor knowledge of the measurements or propagation model or from an approximation of the actual distribution on the impact plane at TCA. As shown in [16] and [17], this epistemic uncertainty can be modelled with DSt.

The idea proposed in [16], was to use DSt to compute the level of confidence in the correctness of the value of the PoC, given the available evidence on the sources of information. Each component of the combined covariance matrix in the impact plane, $[\sigma^{2}_{\xi},\sigma^{2}_{\zeta},\sigma_{\xi\zeta}]$ , was modelled with one or more intervals and so was the miss distance $[\mu_{\xi},\mu_{\zeta}]$ . A basic probability assignment (bpa) was then associated with each interval. The intervals and the associated bpa can be derived, for example, directly from the raw observations, [15, 23], or from a time series of CDMs, [3], as explained later in this paper. Note that in the case in which raw observation data are available, one could directly compute the confidence on the miss distance, see [15]. However, in the following we will consider the CDMs as the observable quantities and the PoC, computed from the CDMs, to be the quantity of interest.

Given the intervals and associated bpa, one can compute the cross-product of all the intervals under the assumption of epistemic independence. Each product of intervals with non-zero bpa constitutes a Focal Element (FE), $\gamma_{i}$ , whose joint bpa is the product of the bpas of the individual intervals. When computing the PoC, each FE defines a family of bi-variate Gaussian distributions on the impact plane. In the following, the collection of all focal elements forms the uncertainty space U, and the uncertain parameter vector is $\mathbf{u}=[\mu_{\xi},\mu_{\zeta},\sigma^{2}_{\xi},\sigma^{2}_{\zeta},\sigma_{% \xi\zeta}]^{T}$ so that $\mathbf{u}\in U$ .

Given the set $\Phi=\{PoC|PoC\geq PoC_{0}\}$ and $\Omega=\{\mathbf{u}\in U|PoC(\mathbf{u})\in\Phi\}$ the Pl and Bel that the $PoC$ is larger than a given threshold $PoC_{0}$ given the available evidence are:

	$Bel(\Omega)=\sum_{\gamma_{i}\subset\Omega}{bpa(\gamma_{i})}$		(2a)
	$Pl(\Omega)=\sum_{\gamma_{i}\cap\Omega\neq\emptyset}{bpa(\gamma_{i})}$		(2b)

For different values $PoC_{0}$ , Eqs. 2a and 2b define two curves (see the example in Fig. 1). The area between the curves, $A_{Pl,Bel}$ , in logarithmic scale, is:

A_{Pl,Bel}=\int_{\log(\underline{PoC})}^{0}Pl(\Omega)\,d(\log(PoC))-\int_{\log% (\underline{PoC})}^{0}Bel(\Omega)\,d(\log(PoC))

(3)

$Bel(\Omega)$ is a lower bound on the probability that $PoC\geq PoC_{0}$ . Its value is computed by adding up all the FEs fully supporting the hypothesis $PoC\geq PoC_{0}$ . $Pl(\Omega)$ is an upper bound on the probability that $PoC\geq PoC_{0}$ . Its value is computed by adding up all the FEs only partially supporting the hypothesis $PoC\geq PoC_{0}$ . The area $A_{Pl,Bel}$ quantifies the amount of uncertainty on the probability that $PoC\geq PoC_{0}$ , i.e. if no epistemic uncertainty is present, both curves would reduced to the same Cumulative Distribution Function (CDF). Thus, for a given value of $PoC_{0}$ , a large value of Pl associated with a small value of $A_{Pl,Bel}$ suggests that there is a lot of support to the hypothesis $PoC\geq PoC_{0}$ given the available information. On the contrary a large value of Pl associated to a large value of $A_{Pl,Bel}$ suggests that the hypothesis $PoC\geq PoC_{0}$ is very plausible to be true but with a high degree of uncertainty.

Refer to caption — Figure 1: Support to the value of $PoC$ being greater than a given value: $Bel$ -black solid line; $Pl$ - black dashed line. The dotted purple line represents a possible $PoC_{0}$ .

In [16], a DSt-based classification system was proposed to decide whether, for a given conjunction event, a Collision Avoidance Manoeuvre (CAM) was required or not. In this paper, we propose a revised version of the classification approach proposed [16]. A given conjunction event is classified according to: i) the value of the Pl at $PoC=PoC_{0}$ or $Pl(PoC_{0})$ , ii) the time of closest approach $t2TCA$ and iii) the area $A_{Pl,Bel}$ . We introduced five thresholds: two time thresholds indicating the proximity of the event, $T_{1}$ and $T_{2}$ , the maximum admissible $PoC$ , or $PoC_{0}$ , the level of Pl, $Pl_{0}$ , above which there is sufficient support to the hypothesis $PoC\geq PoC_{0}$ , and the value of area $A_{0}$ , above which the information is considered to be uncertain. Three of the five thresholds, $T_{1}$ , $T_{2}$ and $PoC_{0}$ , are decided by the operators and depend on operational constraints, the other two $Pl_{0}$ and $A_{0}$ need to be tuned under evidence-based criteria, as it will be explained in the remainder of the paper.

We then introduce the following six classes, see Table 1, each defined by a combination of $Pl(PoC_{0})$ , $t2TCA$ and $A_{Pl,Bel}$ :

Table 1: Conjunction risk assessment evidence-based classification criterion.

Time to TCA	$\mathbf{Pl}$ at $\mathbf{PoC_{0}}$	Area between curves	Class
$t2TCA\leq T_{1}$	$Pl(PoC_{0})<Pl_{0}$	-	5
	$Pl(PoC_{0})\geq Pl_{0}$	$A_{Pl,Bel}<A_{0}$	1
	$Pl(PoC_{0})\geq Pl_{0}$	$A_{Pl,Bel}\geq A_{0}$	0
$T_{1}<t2TCA$ $t2TCA\leq T_{2}$	$Pl(PoC_{0})<Pl_{0}$	-	4
	$Pl(PoC_{0})\geq Pl_{0}$	$A_{Pl,Bel}<A_{0}$	2
	$Pl(PoC_{0})\geq Pl_{0}$	$A_{Pl,Bel}\geq A_{0}$	3
$t2TCA>T_{2}$	-	-	3

•

Class 0: there is enough evidence supporting $PoC\geq PoC_{0}$ but is accompanied by a high degree of uncertainty and no time to acquire new measurements, due to the proximity of the event, hence a CAM is recommended.
•

Class 1: there is full support to the hypothesis $PoC\geq PoC_{0}$ , with limited uncertainty, and short $t2TCA$ , hence a CAM is required.
•

Class 2: there is full support to the hypothesis $PoC\geq PoC_{0}$ , with limited uncertainty, preparing a CAM is recommended, but a CAM is not executed yet due to the available time before the encounter.
•

Class 3: there is enough evidence supporting $PoC\geq PoC_{0}$ but is accompanied by a high degree of uncertainty with sufficient time to acquire new measurements, hence more measurements should be acquired.
•

Class 4: there is insufficient evidence supporting $PoC\geq PoC_{0}$ and sufficient time to acquire new measurements.
•

Class 5: no action is implemented, since $t2TCA$ is too short and there is insufficient evidence supporting $PoC\geq PoC_{0}$ .

Note that for $t2TCA>T_{2}$ all events are classified as Class 3 because the required action is to acquire more measurements. Also, it has to be noted that the level of confidence that one has in the computed value of the $PoC$ depends only on $Pl_{0}$ . If $Pl_{0}$ is set to zero it means that one accepts even a single piece of partial evidence that $PoC\geq PoC_{0}$ to escalate the Class from 5, to 0 or 1, or from 4 to 2 or 3.

3 Modelling Epistemic Uncertainty in Conjunction Data Messages

The use of DSt to model epistemic uncertainty does not require any assumption on the probability of an event and also captures rare events with low probability. On the other hand with no direct information on measurements and dynamic model, one can only rely on the CDMs to define the FEs and associated probability masses.

This section presents a methodology to associate one of the six classes introduced in the previous section to a given sequence of CDMs. The first step is to derive the FEs from the time series of miss distances and covariance matrices in the CDMs. In accordance with DSt, we make no prior assumption on the underlying distribution of the CDMs and, instead, we consider that each CDM is drawn from an unknown set of probability distributions. The assumption is that the value of the uncertain vector $\mathbf{u}$ in each CDMs is a sample drawn from the set of unknown distributions. We make use of the DKW inequality, [20], to build an upper and lower bound to the set starting from the empirical Cumulative Distribution Function (eCDF) derived from the sequence of CDMs.

Given a sequence of CDMs and the eCDF of each of the components of the uncertain vector $\mathbf{u}$ , the DKW inequality defines the following upper and a lower bounds

F_{n}(x)-\sqrt{\frac{\ln{\frac{2}{\delta}}}{2n}}\leq\mathcal{F}(x)\leq F_{n}(x% )+\sqrt{\frac{\ln{\frac{2}{\delta}}}{2n}}

(4)

around the eCDF $F_{n}(x)$ (dashed green lines in Fig. 1(b)), given $n$ CDMs and the confidence level $1-\delta$ that the exact distribution $\mathcal{F}(x)\in F_{n}(x)\pm\varepsilon$ , where $\varepsilon=\sqrt{\frac{\ln{\frac{2}{\delta}}}{2n}}$ .

Note that expression Eq. 4 implies that for an infinite number of observations $\mathcal{F}(x)=F_{n}(x)$ . However, in the following, we will show that in real sequences not all CDMs follow the same distribution. Convergence to a single distribution is, therefore, plausible for a single sequence with consistent measurements and propagation model. Furthermore, $F_{n}(x)$ would converge to a delta function if each observation returned the same mean and covariance and the propagation model would not introduce any variability or nonlinearity.

From the confidence region defined by the DKW bands, it is possible to build a probability box, or p-box, [24, 25, 26], for each of the components of $\mathbf{u}$ . A p-box is a set of all CDFs compatible with the data, that is, the bounded region containing all distributions from where the set of samples may have been drawn, [25]. The upper and lower bounds of the p-box are monotonic non-decreasing functions, ranging from 0 and 1, so that $\underline{\mathcal{F}}(x)\leq\mathcal{F}(x)\leq\overline{\mathcal{F}}(x)$ , with $\underline{\mathcal{F}}(x)$ and $\overline{\mathcal{F}}(x)$ the upper and lower bounds of the p-box for a given variable $x$ , [24].

In this work, the p-box bounds are computed from the CDF of a weighted sum of univariate Gaussians, each one centred at one of the samples. More formally the assumption is that $\mathcal{F}(x)$ can be approximated by:

\mathcal{F}(x)\sim\mathcal{P}(x)=\int_{-\inf}^{\inf}\sum_{i}^{n}w_{i}\mathcal{% N}(x_{i},\sigma_{i};x)\,dx,

(5)

with $x_{i}$ the realisations of the uncertain variable $x$ , $w_{i}$ a weight associated with each sample, and $\sigma_{i}$ the variance of the Gaussian distribution associated with the ith-sample. See Fig. 1(a) for an illustrative example. Implicitly, it implies that each sample presents some uncertainty which is modelled with a Gaussian distribution (grey lines in Fig. 1(a)). This distribution represents the confidence in the sample’s value. By doing so, we admit that when we observe a sequence of CDM we cannot tell from which exact distribution that sequence is drawn. This is consistent with the available sequences of real CDMs and the approach adopted by CNES to model the uncertainty in the covariance realism (see section 4.2.2).

In order to define the limits of the p-box, the two free parameters on each Gaussian distribution on the weighted sum, $w_{i}$ and $\sigma_{i}$ , must be computed by solving the optimisation problems:

\left\{\begin{array}[]{l}\overline{\mathcal{P}}(x)=\max_{w_{i},\sigma_{i}}% \mathcal{P}(x;w_{i},\sigma_{i})\\ \underline{\mathcal{P}}(x)=\min_{w_{i},\sigma_{i}}\mathcal{P}(x;w_{i},\sigma_{% i})\end{array}\right.s.t.\left\{\begin{array}[]{l}\overline{\mathcal{P}}(x)% \leq\min(1,F_{n}(x)+\varepsilon)\\ \underline{\mathcal{P}}(x)\geq\max(0,F_{n}(x)-\epsilon)\end{array}\right.\text% {,}

(6)

where $\overline{\mathcal{P}}(x),\underline{\mathcal{P}}(x)$ are the upper and lower bounds of the p-box, respectively (red dashed-pointed line in Fig. 1(c)). An approximation to $\overline{\mathcal{P}}(x),\underline{\mathcal{P}}(x)$ can be computed by finding the values of $w_{i}$ and $\sigma_{i}$ in Eq. 5 that best fit the upper and lower DKW bands:

\left\{\begin{array}[]{l}\overline{\mathcal{P}}(x)\approx\overline{P}(x)=\text% {fit}_{w_{i},\sigma_{i}}(F_{n}(x)+\epsilon)\\ \underline{\mathcal{P}}(x)\approx\underline{P}(x)=\text{fit}_{w_{i},\sigma_{i}% }(F_{n}(x)-\epsilon)\end{array}\right.\text{.}

(7)

Eq. 7 gives the upper and lower bounds on the probability of realising a particular value of the uncertain vector $\mathbf{u}$ but the definition of a set of intervals for each component of $\mathbf{u}$ requires first the definition of the range of each component. Eq. 5 suggests that each p-box has infinite support. However, this would lead to an inconvenient infinite range for variance and miss distance. Instead, in the following we define the more practical interval $[\underline{x},\overline{x}]$ such that:

\int_{\underline{x}}^{\infty}w_{1}\mathcal{N}(x_{1},\sigma_{1};x)\,dx=0.99,\;% \;\int_{-\infty}^{\overline{x}}w_{n}\mathcal{N}(x_{n},\sigma_{i};x)\,dx=0.99.

(8)

It is important to note that the assumption is that the miss distance and each component of the covariance can be treated independently. This is generally not the case, however, the independence assumption in this paper leads to a more conservative set of focal elements that cover the space of realisations of the uncertainty vector. Although this can lead to over-conservative decisions, it is deemed to be acceptable in the case of high-risk events with little available information.

3.1 Scaling of the CDMs

The approach described in previous sections assumes that every CDM has the same relative importance and no additional source of information is available to qualify each individual CDM. However, as the $t2TCA$ decreases, so does the effect of the uncertainty on the true shape of the distribution on the impact plane and the effect of model uncertainty in the propagation. Fig. 2(a) shows the normalised determinant of multiple sequences of covariance matrices taken from the database of the ESA’s Collision Avoidance Kelvins Challenge, [27, 8]. The database contains 13,152 sequences of CDMs of some of the Low Earth Orbit (LEO) satellites monitored by the ESA Space Debris Office (SDO). The figure shows that one can fit the simple exponential law $y^{\prime}=e^{-3t^{\prime}}$ to the magnitude of the determinant (red line in the figure). However, one cannot simply trust later CDMs due to large uncertainty in each individual sequence. Thus, we propose the following fit for each individual sequence:

	$y^{\prime}=Ce^{At^{\prime}}+B\hskip 15.0ptA,B,C\geq 0,$		(9a)
	$y^{\prime}=\frac{\\|\mathbf{\Sigma}\\|}{\max_{CDMs}(\\|\mathbf{\Sigma}\\|)}$		(9b)
	$t^{\prime}=\frac{(1-\max_{CDMs}(t2TCA))}{(\min_{CDMs}(t2TCA)-max_{CDMs}{t2TCA})}$		(9c)

Once the parameters $A$ , $B$ and $C$ are fitted to the samples from a given sequence, the following weight is associated with each CDM in that sequence:

w_{CDM_{i}}=\frac{1}{y^{\prime}(t2TCA_{CDM_{i}})}

(10)

The weight is applied to each sample in the eCDF used to compute the DKW bounds: the probability mass associated with each sample is re-scaled by a factor $w_{CDM_{i}}$ . See Fig. 4 where the eCDF of $\mu_{\xi}$ for an example with 5 observations is shown both with samples equally weighted (dashed red) or having applied the weighting law described above (blue).

This approach results in a scaling of the probability mass associated with the CDMs but still allows the quantification of highly uncertain CDMs since there is no filtering process. The reason is that, with no information on trusted sources or individual CDMs, one cannot make any assumption on which CDM is more credible.

3.2 $\alpha$ -cuts and DSt Structures

Once a p-box is defined, the intervals for each component of $\mathbf{u}$ are derived from a series of equally spaced $\alpha$ -cuts, light blue and grey dotted horizontal thin lines in Fig. 1(d). Each $\alpha$ -cut creates interval, [28, 29]:

[x_{\alpha},x^{\alpha}]=\left\{x\,|\,\mathcal{F}(x)\geq\alpha\right\}\text{.}

(11)

The intersection with the upper bounds in the p-box defines the lower limit of the interval, and the intersections with the lower bound define the upper limit of the interval. The number of intervals is equal to the number of cuts plus one, and the bpa associated with each interval, assuming the cuts are evenly spaced, is equal to the inverse of the number of cuts. The intervals and their bpa will define an envelope around the p-box (blue and black dashed lines in Fig. 1(d)). The greater the number of $\alpha$ -cuts, the closer the envelope will be to the p-box, but the more computationally expensive is the computation of Bel and Pl. From the intervals associated with each component of $\mathbf{u}$ one can compute the FE $\gamma_{i}$ and their associated $bpa(\gamma_{i})$ by performing the Cartesian product of all the intervals and associated bpas. Once the FE and bpas are computed, the Pl, Bel of $PoC\geq PoC_{0}$ are computed with Section 2 (see Fig. 5) and the conjunction event is classified according to Table 1.

Even in this case, we implicitly maintained the assumption that variables are independent, although it is not true that the components of the miss distance and of the covariance are all independent. Approaches to address dependencies already exist in the literature, see [30], and will be considered in future works. The independence assumption has two implications: i) the uncertainty space $U$ is an outer approximation of the space of all distributions of $\mathbf{u}$ and ii) some focal elements might not contain any sample of $\mathbf{u}$ . The combination of the two generally leads to over-conservative results. Thus, in order to partially recover the interdependence between uncertain quantities, yet coherent with DSt, a $bpa=0$ is assigned to all empty FEs and their bpa, coming from the Cartesian product, is evenly distributed to the rest of FEs so that $\sum_{i}bpa(\gamma_{i})=1$ .

4 Numerical experiments

In this section, some numerical tests are presented. The aim is to show the applicability of the methodology presented in previous sections and compare its outcome to the decisions made in past real cases by actual satellite operators: European Space Operations Centre (ESOC) and CNES.

4.1 Parameter Tuning

The methodology proposed in this paper requires the prior definition of the values of two thresholds: $Pl_{0}$ and $A_{0}$ . These two thresholds should be tuned by analysing a large dataset of conjunction events with known outcomes. However, in every database of CDMs available to the authors, the number of provable Class 1 and 2 conjunctions is very small or zero.

Since $A_{0}$ does not affect Class 4 and 5, which depend only on $Pl_{0}$ , but influences the number of True Positives (actual collisions) and False Positives (no-collisions believed to be collisions), one can define $Pl_{0}$ first and then use $A_{0}$ to quantify the degree of uncertainty in the class associated to an event. According to the classification in Table 1, the expected outcome is that low values of $Pl_{0}$ would increase the number of events classified as Class 1 or 2, reducing, at the same time, the number of False Negatives (collisions believed to be no-collisions) and increasing the amount of TPs. If this is combined with high values of $A_{0}$ , the chances of detecting all high-risk events are high, but at the cost of increasing the number of FPs. If instead, $A_{0}$ is low, more events will be classified as uncertain (Class 0 and 3). On the contrary, a higher value of $Pl_{0}$ would reduce the false alerts, FPs, but at the risk of increasing the number of FNs.

This paper used the DSt structure to set a value for $Pl_{0}$ . If there is at least one FE supporting $PoC>PoC_{0}$ , it means that there exists at least one piece of evidence suggesting that the PoC can be correct. This piece of evidence may correspond to an extreme event with low probability. Following this idea, we propose the value $Pl_{0}=\min_{i}(bpa(\gamma_{i}))$ . This implies that even a PoC that corresponds to a rare event in the generation of a CDM is considered to be plausible. The value of $A_{0}$ is selected by balancing the number of TPs and FPs. The idea is to try to reduce the number of FPs by reclassifying them as uncertain cases and presenting the level of such uncertainty to the operator. A low value of $A_{0}$ implies that the operator accepts very little uncertainty in the sequence of CDM, which reduces the number of FPs but potentially classifies some TPs as uncertain. On the other hand, a greater value of $A_{0}$ implies that the operator is very conservative and accepts to treat a number of FPs as TPs. Thus, the decision to execute a CAM is related to the confidence of the operator in the quality of the CDMs. For highly uncertain sequences of CDMs, a low $A_{0}$ is recommended, but if the quality of the CDMs is high, a higher $A_{0}$ should be used.

In the following, rather than selecting the value of the area threshold $A_{0}$ , we select the value of the normalised area $A^{*}_{0}=\{0,0.05,0.1,0.15,...,0.95,1\}$ , where $A^{*}_{0}$ is the fraction of the maximum possible area between the $Bel$ and $Pl$ curves, that is, when Bel drops to zero at the minimum value of PoC, $\underline{PoC}$ , and Pl remains equal to one until $PoC=1$ . In this tuning exercise the area is computed by taking the lower limit $\underline{PoC}=10^{-30}$ for the PoC as this is the lowest value computed from all the sequences of CDMs in our database. For all the first four tests in this paper, we will use a value of $A_{0}^{*}=0.1$ that allows one to clearly differentiate Event 1 from Events 3 and 4 in the following section. In the last test, we will present the sensitivity of the number of recommended CAMs to the value of $A_{0}^{*}$ .

4.2 Comparison Against SDO and CNES

The results in this section will show a comparison between the CARA performed with the proposed evidence-based method and the decisions made by real operators in a selected number of real cases. The two operators considered in this study are the ESA’s SDO and CNES. Each of them has a different approach to conjunction analysis. Four real conjunction events are analysed and the different operational approaches are compared.

For all examples the values of the thresholds are reported in in Table 2.

Table 2: Threshold values.

Threshold	Units	Value
$T_{1}$	days	3
$T_{2}$	days	5
$PoC_{0}$	-	$10^{-4}$
$Pl_{0}$	-	$1/\#FE$
$A^{*}_{0}$	-	$0.1$
$\underline{PoC}$	-	$10^{-30}$

The evolution of the normalised area gap between the $Pl$ and $Bel$ curves, or $A_{Pl,Brel}$ , over time, for all four cases can be found in Figure 6, where $A^{*}_{Pl,Bel}=A_{Pl,Bel}/max(A_{Pl,Bel})$ is the normalised area between curves, $A_{Pl,Bel}$ , defined in Eq. 3. The Figure confirms that an $A^{*}_{0}=0.1$ is appropriate to differentiate between cases like Event 1 from cases like Event 3 and 4. All four cases are presented in more detail in the following subsections.

4.2.1 Space Debris Office Conjunction Risk Assessment

The approach followed by the ESA SDO is probability-based, relying mainly on the value of the PoC computed with the information from the CDMs, or the PoC included in the CDM. The following quote may summarise the conjunction risk assessment process adopted by the SDO: \sayFor a given close approach the last obtained CDM, including the computed risk, can be assumed to be the best knowledge we have about the potential collision and the state of the two objects in question. In most cases, the Space Debris Office will alarm control teams and start thinking about a potential avoidance manoeuvre 2 days prior to the close approach in order to avoid the risk of collision, to then make a final decision 1 day prior, [27]. Nevertheless, each mission monitored by the SDO has specific operational constraints (i.e. the time needed to prepare and execute the manoeuvre) and will have its own risk and time thresholds, $PoC_{0}$ and $T_{1}$ . The time threshold $T_{1}$ is generally 2 or 3 days away from TCA. At that point the mission team is informed about the possible collision, and a final decision is usually made (when possible) 1 day from TCA, [27]. The risk threshold $PoC_{0}$ is determined statistically based on the overall collision risk and the annual frequency of close approaches, trading off the ignored risk and the avoided risk by selecting the risk threshold at the cost of an expected number of annual manoeuvres, see [1]. Generally, for missions in the LEO regime, a threshold of $PoC_{0}=10^{-4}$ leads to a risk reduction of around 90% at the expense of 1 to 3 manoeuvres per year, with current levels of traffic. However, a lower threshold, around $10^{-5}$ , may be considered to ensure sufficient time to prepare a collision avoidance manoeuvre in the case of escalated events, [1].

Following this approach, the SDO escalates an event when the PoC of the last CDM is bigger than the threshold. Escalating an event means that further and more detailed analyses are required. If the risk is still above the threshold at the decision time, a CAM is designed in cooperation with the mission team, whose final decision will be made based on the value of PoC included in the last CDM received before the go/no-go decision time. More detailed information on the CARA process of the SDO can be found in [1]. For the first three events in this subsection, only CDMss from the MiniCat database were considered.

Event #1

This event represents a high-risk scenario provided by the ESA SDO. The uncertain geometry in the impact plane, with the whole sequence of CDMs and the PoC evolution are displayed in Fig. 7. Events with PoC above the threshold for times to TCA greater than $T_{1}$ make the event escalate, that is, they are further analysed and possible alerts to the mission’s team can be triggered, while high-risk CDMs received in the last 72 hours trigger a CAM procedure.

From Fig. 6(b), one can see that the PoC remains high along the whole sequence. Even if at the beginning it was below the threshold, its proximity to $PoC_{0}$ along with the upward trend made the operator escalate the event. The PoC threshold was violated within the last few days before TCA, which led to a CAM execution to reduce the risk of the event.

We applied our evidence-based methodology to this case by following the approach presented in Section 3. The DKW bands were computed assuming a confidence interval $\delta=0.5$ . The CDMs were weighted according to the exponential law in Section 3.1. Fig. 8 shows the fitting law after having received all the CDMs (red) along with the value of the combined covariance matrix determinant, for the whole sequence (black). For the fitting law in Fig. 7(b), the value of the dimensionless parameters in Eq. 9a after having received the whole sequence are: $A=1.0752,B=0.9811,C=0.001716$ . Note that the value of the parameters varies with the number of CDMs received to better fit the covariance determinant evolution up to that time.

We repeated the same analysis with different numbers of $\alpha$ -cuts per uncertain variable: $\#\alpha\text{-cuts}=\left\{1,2,3,4,5,7\right\}$ . These cuts led to a number of intervals per variable equal to $\#\text{intervals}=\left\{2,3,4,5,6,8\right\}$ , which translated into a number of FEs $\#\text{FE}=\left\{32,243,1024,3125,7776,16807\right\}$ , respectively. The Pl and Bel curves for the PoC, for each number of cuts, is presented in Fig. 9, after having received the whole sequence of CDMs.

Fig. 9 shows that, although the increasing number of $\alpha$ -cuts provides a more refined set of curves, their shape and values varies only slightly. In this case, the Bel and Pl curves overlap for most values of PoC except for a small interval around the $PoC_{0}$ , as it could be expected both, from the uncertainty geometry in Fig. 6(a) and the values of the PoC in Fig. 6(b). Since the information in the CDM is coherent across the whole sequence, the gap between the Pl and Bel curves is small.

Fig. 10 shows the classification, purple solid line, as a function of the time to the TCA from the last received CDM. The figure shows also the PoC directly computed from the CDM.

Initially, the event is classified as Class 4 and rapidly falls to Class 5, since there is little evidence supporting a higher PoC. However, at 2.5 days from TCA, the PoC consistently grows above the threshold. Given the little uncertainty in the sequence of CDMs the event is reclassified as Class 1 and a CAM is recommended.

This is the same decision finally taken by the SDO. As seen in Fig. 9, the support for a high value of PoC is high and the gap between the curves (level of uncertainty) is very small. Thus, the recommended action in the last days prior to the encounter would be to implement a manoeuvre to reduce the risk of a collision.

Event #2

A similar analysis was done for the Low-risk conjunction event illustrated in Fig. 11, also provided by the ESA SDO. Opposite to the previous event, in this case, the PoC remains well below the threshold, so no alert is required to be triggered and no CAM is required to be designed or executed.

The evidence-based analysis was performed using the same parameters as before: $\delta=0.5$ for the DKW bands, with a different number of $\alpha$ -cuts: $\#\alpha\text{-cuts}=\left\{1,2,3,4,5,7\right\}$ per variable. The final set of CDMs was weighted with the exponential fitting law Eq. 9a using the following parameters: $A=0.6049,B=5.0896,C=0.4518$ . The fitting law (red) and the combined covariance matrix determinant in the CDMs (black) appear in Fig. 12. Note the convergence in the second half of the sequence.

In Fig. 13, the corresponding Pl and Bel curves on the value of PoC after having received all the CDMs of the event are shown. Again, increasing the number of $\alpha$ -cuts makes the curves smoother and shows a converging trend, but does not change the overall confidence in the value PoC. The maximum value of PoC with some supporting evidence is well below the threshold, indicating that the event can be deemed to be safe. However, the left-most part of the Bel and Pl curves shows a significant gap. This can be explained by the fact that the ellipses are not too different from each other (Fig. 10(a)) and they tend to converge to a single ellipse for the later CDMs, as shown in Fig. 10(b). Thus, the initial information content in each CDMtends to support lower values of PoC, which explains the lower value of Bel on the left of the graph. However, due to the concentration of information around the later CDMs, the big drop both in Pl and Bel occurs at PoC $\sim 10^{-7}$ .

Finally, the conjunction assessment for the whole sequence is shown in Fig. 14. This event displays a greater uncertainty with respect to the previous scenario, but values of the PoC greater than $10^{-7}$ have no supporting evidence and $Pl=Bel=0$ . Thus, the event is initially classified as Class 4 ( $t2TCA>T_{1}$ ) and then dropped to Class 5 ( $t2TCA\leq T_{1}$ ) for the whole sequence, meaning that no further action should be taken by the operator. This is the same decision made by the SDO.

Event #3

This last event is affected by a significant level of uncertainty. The encounter geometry and the evolution of the PoC in the CDMs are shown in Fig. 15. Despite the initial high risk, with values of PoC close to the threshold, the final decision of the SDO was not to take any further action. This decision was dictated by the later values of the PoC, that were all consistently lower than the initial ones, and considerably below $PoC_{0}$ .

The evidence-based analysis was performed with the same parameters as before: $\delta=0.5$ for the DKW bands. The exponential fitting law Eq. 9a to weight the CDMs, after having received the whole sequence, had the following parameters $A=0.7917,B=7.1471,C=0.1858$ and is shown in Fig. 16 (red line) along with the covariance matrix determinant (black line).

The Pl and Bel curves for the PoC were computed for different $\alpha$ -cuts: $\#\alpha\text{-cuts}=\left\{1,2,3,4,5,7\right\}$ . The curves are shown in Fig. 17. In this case, there is a significant gap between Pl and Bel for all the values of PoC for which $Pl>0$ . This uncertainty (or level of disagreement between CDMs) can be seen in Fig. 14(a), which shows the variety of the uncertainty ellipses from the beginning of the sequence to the last CDMs. In this case the supporting evidence that a value of $PoC>PoC_{0}$ is plausible does not go to zero but the gap between the Pl and Bel curves suggests that a further analysis is required although the value of Pl is low and Bel is zero.

Fig. 18 shows the result of the classification: the event starts at Class 2, given the potential high risk suggested by the initial CDMs but quickly drops to Class 3 ( $t2TCA>T_{1}$ ) because of the level of uncertainty and is finally classified as Class 0 (for $t2TCA\leq T_{1}$ ). In this case, our approach would suggest a further analysis due to the non-zero plausibility of a high PoC and a high difference between Pl and Bel, while the decision made by the SDO was to take no further action. The more prudent recommendation coming from our classification system would lead to a further inspection of the Pl curve with the realisation that the supporting evidence is small, albeit not zero.

4.2.2 CNES Conjunction Risk Assessment

In order to compensate for the possible lack of realism of the covariance matrix at TCA, CNES re-scales both the covariance matrix of the primary and secondary body with two factors, respectively $k_{p}\in K_{P}$ and $k_{s}\in K_{S}$ . A scaled PoC, called scaled Probability of Collision (sPoC), is obtained by solving the following PoC maximisation problem (see [10, 11]):

\left\{\begin{array}[]{l}sPoC=\max_{k_{p}\in K_{P},k_{s}\in K_{S}}PoC(\Sigma)% \\ \text{s.t.}\hskip 10.0pt\mathbf{\Sigma}=k_{p}^{2}\mathbf{\Sigma}_{p}+k_{s}^{2}% \mathbf{\Sigma}_{s}\end{array}\right.,

(12)

where $\mathbf{\Sigma}_{p}$ and $\mathbf{\Sigma}_{s}$ are, respectively, the primary and secondary covariance matrices in a given CDM associated to the conjunction event under consideration.

The two sets $K_{P}$ and $K_{S}$ are derived, for each sequence of CDMs, under the assumption that CDMs are samples drawn from an underlying distribution, and the last CDM contains the most reliable estimation of the position of the two objects. Thus, by using the last CDMs as a reference, it is possible to compute the Mahalanobis distance of all previous CDMs from the last one. If one assumes that the uncertainty in position is Gaussian, the Mahalanobis distance should follow a $X^{2}$ distribution with 3 degrees of freedom. By performing a Kolmogorov-Smirnov (KS) test between the distribution of the computed Mahalanobis distances and the theoretical one, and setting a desired level of realism, one can define the sets $K_{P}$ and $K_{S}$ . More details can be found in [31].

CNES decision-making is based on both the value of sPoC and a number of geometric considerations. Events with values of $sPoC>5\cdot 10^{-4}$ are classified as High-Interest Event, the more risky classification level (red level). For values of $10^{-4}<sPoC<5\cdot 10^{-4}$ , the event is classified as an Interest Event, the second level of risk (orange level). If the value of the $sPoC$ is below those thresholds, caution geometric criteria are applied: miss distance below 1 km or radial distance below 200 m. Note that these threshold values are the default ones and may differ from mission to mission. If the CDMs are received 4-5 days before the encounter or earlier, no alerts are raised independently of the value of sPoC, although the event is placed under study if some of the above criteria are violated. For later CDMs, alerts may be raised according to the value of sPoC. Finally, if the high risk continues after the decision time (usually 2 days before the encounter), a final decision is made before the TCA.

In the following, we will test our approach on a real close encounter faced by CNES and compare our classification against the one of CNES.

Event #4

This scenario presents a high-risk collision case for a real close encounter where CNES had to implement a manoeuvre to reduce the risk.

Fig. 18(a) shows the geometry of the event, where the earlier CDMs (green ellipses) suggested a low PoC, while later CDMs (red and amber ellipses) suggest a high PoC. Fig. 18(b) shows the PoC and the sPoC. The latter is above the threshold $10^{-4}$ from the start and progressively increases while the PoC displays a large variability till about a day before TCA. CNES classified the event as High-Interest Event, meaning that careful monitoring was required, starting from the 12 ${}^{th}$ CDM (2.96 days before the TCA). The final decision to perform a manoeuvre was taken 30 hours before the encounter. Note that the CDM received about a 1.2 days from TCA indicates a PoC $<10^{-5}$ , well below the risk threshold, while the sPoC indicates a risk above $10^{-3}$ , which aligns better with the last three CDMs received between the decision time and the CAM execution time).

The evidence-based analysis was performed following the same approach as for the SDO cases, with $\#intervals=\left\{2,3,4,5,6,8\right\}$ intervals per variable and CDM weighted according to the exponential law in Fig. 20.

The Pl and Bel corresponding to the whole sequence of CDM are shown in Fig. 21, and the classification sequence for different numbers of intervals is shown in Fig. 22. In Fig. 21 one can see that $Pl(PoC_{0})$ is nearly 1, and $Pl(sPoC)>0$ along the whole time series. In fact, $Pl=0$ at $PoC\sim 10^{-2}$ , while $\max(sPoC)=5\cdot 10^{-3}$ . However, the gap between the Pl and Bel curves is very high, indicating a degree of uncertainty in the sequence of CDMs. This is due to the variability in the CDMs. Thus the event is classified as Class 0.

Although this event is placed in the same class as Event 3, the supporting evidence is quite different. Event 4 has a $Pl\approx 1$ and Bel different from zero at $PoC_{0}$ while Event 3 has $Bel=0$ and $Pl<0.2$ at $PoC_{0}$ . This means that, although in this paper we opted for a very conservative classification of the events such that both Events 3 and 4 fall in the same uncertainty class, a simple analysis of the Bel and Pl curves would suggests that the available evidence for Event 4 supports a high probability of collision, up to $10^{-2}$ in fact, while for Event 3 the supporting evidence at $PoC_{0}$ is quite low.

4.3 Statistical Analysis of CAM Executions

After having compared the proposed evidence-based conjunction assessment approach against real operations on specific cases, in this section we compare the number CAMs that our evidence-based approach would recommend over a large number of real conjunctions experienced by a single mission.

The selected mission is the ESA SWARM-A satellite, orbiting in the LEO regime (circular polar orbit of 87.7 deg at 511 km of altitude), dedicated to studying the Earth’s magnetic field as part of a constellation of three satellites. The mission thresholds to trigger conjunction alerts are $PoC_{0}=10^{-4}$ and $T_{1}=72$ hours. Thus, any satellite with a PoC above the threshold in the last 3 days would escalate and would require further analysis, and eventually, a possible CAM design or execution. Nevertheless, encounters presenting a higher risk or an increasing trend before $T_{1}$ may be escalated if the operator considers that there is a potential risk for the mission. Finally, the go/no-go decision is subject to operational constraints: the time required to design a CAM after receiving the triggering manoeuvre, the possibility to upload and check the design manoeuvre and the ground station availability.

The database of CDMs includes alerts from 2015 to 2022, with a total of 36,072 events. Overall, most of the events in the database did not represent a threat to the satellite, with only 20 representing escalated events. As explained before, an escalated event is an encounter where the PoC, or the PoC trend, suggests that the conjunction may be high risk. From those escalated events, only 2 required a CAM to be executed.

The evidence-based analysis was performed with the same thresholds as the previous study cases (Table 2): $PoC_{0}=10^{-4}$ , $T_{1}=3$ days, $T_{2}=5$ days, $Pl_{0}=1/243$ , $A^{*}_{0}=0.1$ , with $\underline{PoC}=10^{-30}$ , and $A_{0}=3$ . The DKW bands were obtained assuming a confidence interval of $\delta=0.5$ . As shown before, a higher number of $\alpha$ -cuts would refine the Pl and Bel curves, providing closer curves that better represent the actual epistemic uncertainty. However, this is at the expense of increasing the computational cost and with limited impact on the final classification. Thus 2 $\alpha$ -cuts (3 intervals) per variable, with a total of 243 FEs per analysis were used.

Since the evidence-based analysis lacks the real information available in the actual operation of the satellite that may have affected the operator decision (for example, the ground station availability or the mission constraints), the statistics were computed at four decision times: $T_{d}=3$ days to the TCA, corresponding with the mission time threshold, $T_{1}$ ; $T_{d}=2$ days to the TCA, allowing for more data to arrive; $T_{d}=1$ day to the encounter, the usual go-no go decision time in ESA’s missions, [1]; and the epoch of the last CDM in the sequence, $T_{d}=0$ . For simplicity, we assume that there is no operational constraint that prevents or modifies the final decision and all information is, thus, available.

Table 3 includes the results from the analysis, compared with the actual statistics provided by the SDO. It is important to bear in mind the differences between the approaches. An event classified as Class 3 or Class 0 (labelled as Uncertain), with the evidence-based approach, would not correspond, necessarily, to an escalated event, since the meaning is different: while an escalated event assumes a certain level of risk, a Class 0 or 3, suggests a degree of uncertainty that requires further investigation before making a final decision. This further investigation might be simply limited to an inspection of the Bel and Pl curves as in cases 3 and 4 above or might require additional observations. On the other hand, for all Class 1 events, the recommendation is to perform a CAM.

Table 3: Results from the statistical analysis on the SWARM-A mission, with the SDO approach and the evidence-based approach. Threshold:

PoC_{0}=10^{-4}

T_{1}=3

days,

T_{2}=5

days,

Pl_{0}=1/243

. Partition with 2

\alpha

-cuts per variable. Upper tier:

A^{*}_{0}=0.1

(

A_{0}=3

); middle tier:

A^{*}_{0}=0.5

(

A_{0}=15

); lower tier:

A^{*}_{0}=0.8

(

A_{0}=24

SDO		Evidence-based
# events		$A^{*}_{0}$	# events	$T_{d}=3$	$T_{d}=2$	$T_{d}=2$	$T_{d}=0$
Total	36,072		Total	24,296	27,918	32,108	36,072
Escalated	20	0.1	Unc.	120	130	172	293
CAM	2		CAM	1	2	3	2
		0.5	Unc.	102	98	107	154
			CAM	19	34	68	141
		0.8	Unc.	95	83	77	75
			CAM	26	49	98	220

From the upper tier in Table 3 (with $A^{*}_{0}=0.1$ ), one can observe that: i) the total number of events increases with the delay in the decision time because more CDMs are available for a decision; ii) the number of manoeuvres proposed by the evidence-based approach is similar to the number of CAMs proposed by the SDO operators; iii) the evidence-based classification system found many more uncertain cases than the SDO. The Table shows also the number of CAMs and uncertain events for $A^{*}_{0}$ equal to 0.5 and 0.8. As expected, an increase in the values of $A^{*}_{0}$ increases the number of CAMs and reduces the number of uncertain cases.

Even if the $Pl_{0}$ threshold is quite low, the number of events escalating to Class 1 remains small. Thus, in this test case, the system is robust enough to remove false negatives without introducing false alerts. Also, the number of CAMs remains roughly constant independently of the decision time (especially, for the selected default value of $A^{*}_{0}=0.1$ ). On the other hand, the number of Class 0 events is between 6 and 15 times higher than the number of escalated events proposed by ESOC. It is here where the evidence-based system differentiates from the probabilistic approach used by ESOC. Class 0 events are those with $Pl(PoC_{0})>Pl_{0}$ , but are still deemed uncertain because $A_{Pl,Bel}>A_{0}$ . $Pl$ captures all realisations, within each Focal Element, that correspond to extreme cases, extreme low or extreme high $PoC$ , compatible with the observed sequence of CDMs. Hence, a large $A_{Pl,Bel}$ with high $Pl$ signifies that there is the evidence that a high $PoC$ event can occur but is uncertain. As in the case of Event 3, many of these cases display a low Pl and zero Bel. Others present conflicting CDM, that cannot be resolved without further observations, or a high Pl for high PoC values, as in Event 4 but with a low Bel. An example can be seen in Figure 23. The evolution of the combined covariance shows a radical rotation of nearly 90 degrees at -4 days from TCA. The evolution of the PoC does not provide any evidence that the covariance had a step change, but remains close to the threshold limit. The evidence-based approach, instead. shows quite some uncertainty and maintains a high $Pl$ till the end of the sequence, suggesting that the event cannot be discarded and requires further analysis.

Note that the percentage of events in this category increases when delaying the decision. This indicates a growing disagreement among CDMs in the sequence as the time approaches TCA, an aspect usually overlooked by probabilistic-based approaches.

5 Conclusions

This work presented a methodology to model and quantify the epistemic uncertainty in a sequence of CDMs, and exploit this quantification to make robust decisions about conjunction events. The method was tested against real operations on a number of real scenarios. The key working assumption was that the value of the miss distance and covariance matrix in each CDM were drawn from a set of unknown distributions. The DKW inequality was used to build bounds on this set and derive a set of focal elements, with associated probability mass supporting a given value of the probability of collision.

The collection of focal elements was used to compute the Pl and Bel on a given value of the PoC. The Pl at $PoC_{0}$ , or $Pl_{0}$ was proposed as a further criterion to make a decision on the actual severity of a conjunction event, while the difference between Pl and Bel, or $A_{Pl,Bel}$ , was proposed as a measure of the uncertainty in the quantification of the PoC.

It was found that when the set of CDMs contains coherent information over the whole time series, the proposed classification system suggests the same decisions normally made by the ESA SDO. When the sequence of CDMs presents a higher degree of variability or a degree of inconsistency the proposed evidence-based approach recommends more conservative decisions compared to the SDO but also provides the operator with a quantification of the related uncertainty.

A comparison with the approach used at CNES, based on the concept of sPoC, showed that the proposed evidence-based approach returns decisions that are less conservative but, at the same time, provides a higher level of information on the uncertainty in the decision. By comparing the ESA and CNES uncertain cases, it was also found that a further inspection of the Pl and Bel curves offers a way to disambiguate the events as the different evolution of PoC over time is reflected in a lower or higher value of Pl and Bel.

Finally, a statistical analysis on a database of real encounters of an ESA mission showed that the number of recommended CAMs is similar but the evidence-based approach tends to detect a higher number of uncertain cases that require further analysis.

Although in our analysis no operational constraints were considered, the number of detected uncertain cases suggests that relying only on the last CDM may be too optimistic while the scaled PoC approach might be too pessimistic without a further uncertainty quantification. In relation to the uncertain cases, different situations can be found which may lead the operator to take different actions. Further analysis on the treatment of these scenarios should be taken and a threshold tuning analysis using virtual datasets or a mixed dataset of real and virtual CDMs may help with this task. The approach proposed in this paper assumes that no additional information on the CDMs is available nor that information on the uncertainty in the propagation model or individual observations can be used. However, if additional information was available one could improve the quantification of uncertainty of each CDMs and build better defined p-boxes with tighter bounds.

Future work will need to consider the correlation and interdependence among variables during the construction of the focal elements and build a more refined model. Furthermore, the current databases of real CDMs do not represent a controlled set of events, because the actual outcome is unknown. A representative synthetic database would greatly help in improving the classification system. Last but not least, machine learning can be used to directly classify events from the time series of CDMs. This approach represents an extension of what was already proposed by the authors and would improve on current efforts to predict the last CDM with machine learning as it would embed a quantification of uncertainty in the prediction.

Acknowledgements

This work was funded by the European Space Agency, through the Open Space Innovation Platform (OSIP), "Idea I-2019-01650: Artificial Intelligence for Space Traffic Management".

The authors would like to thank CNES-Toulouse and the DOA/SME/SE Office for the opportunity to research with them and for sharing really valuable information with us. More specifically, we would like to thank François Laporte for the insightful discussions during the approach development. The authors would like to thank the ESA’s Space Debris Office at ESOC for providing both very useful data and feedback on this work during the research stays at their facilities.

References

Merz et al. [2017] Merz, K., Braun, V., Benjamin Bastida, V., Flohrer, T., Funke, Q., Krag, H., and Lemmens, S., “Current collision avoidance service by ESA’s Space Debris Office,” 7 ${}^{\text{th}}$ European Conference on Space Debris, ESA/ESOC, Darmstadt, Germany, 2017.
Newman et al. [2019] Newman, L. K., Mashiku, A. K., Hejduk, M. D., Johnson, M. D., and Rosa, J. D., “NASA Conjunction Assessment Risk Analysis (CARA) updated requirements architecture,” AAS/AIAA Astrodynamics Specialist Conference, Portland, Maine, US, 2019.
CCSDS [2013] CCSDS, “Recommended Standard: CCSDS 508.0-B-1. Recommendation for space data system standards. Conjunction data message,” Tech. rep., CCSDS, Washington, DC, USA, Jun. 2013. https://public.ccsds.org/Pubs/508x0b1e2s.pdf.
Aristoff et al. [2014] Aristoff, J. M., Horwood, J. T., Singh, N., and Poore, A. B., “Nonlinear uncertainty propagation in orbital elements and transformation to Cartesian space without loss of realism,” AAS/AIAA Astrodynamics Specialist Conference, San Diego, CA, US, 2014.
Cano et al. [2023] Cano, A., Pastor, A., Escobar, D., Míguez, J., and Sanjurjo-Rivo, M., “Covariance determination for improving uncertainty realism in orbit determination and propagation,” Advances in Space Research. Space Environment Management and Space Sustainability, Vol. 72, No. 7, 2023, pp. 2759–2777. DOI: https://doi.org/10.1016/j.asr.2022.08.001.
Pinto et al. [2020] Pinto, F., Acciarini, G., Metz, S., Boufelja, S., Kaczmarek, S., Merz, K., Martínez-Heras, J., Letizia, F., Bridges, C., and Baydin, A., “Towards automated satellite conjunction management with bayesian deep learning,” AI for Earth Sciences Workshop at NeurIPS, 2020. URL https://nips.cc/virtual/2020/public/workshop_16105.html.
Acciarini et al. [2021] Acciarini, G., Pinto, F., Letizia, F., Martínez-Heras, J. A., Merz, K., Bridges, C. P., and Güneş Baydin, A., “Kessler: a machine learning library for spacecraft collision avoidance,” 8 ${}^{\text{th}}$ European Conference on Space Debris, ESA/ESOC, Darmstadt, Germany, 2021.
Uriot et al. [2022] Uriot, T., Izzo, D., Simões, L. F., Abay, R., Einecke, N., Rebhan, S., Martinez-Heras, J. A., Letizia, F., Siminski, J., and Merz, K., “Spacecraft collision avoidance challenge: design and results of a machine learning competition,” Astrodynamics, Vol. 6, No. 2, 2022, pp. 121–140. DOI: https://doi.org/10.1007/s42064-021-0101-5.
Caldas et al. [2023] Caldas, F., Soares, C., Nunes, C., and Guimarães, M., “Conjunction Data Messages for space collision behave as a Poisson process,” 31 ${}^{\text{st}}$ European Signal Processing Conference (EUSIPCO), Hesinki, Finland, 2023.
Laporte [2014a] Laporte, F., “JAC Software, dedicated to the analysis of conjunction messages,” SpaceOps 2014 Conference, Pasadena, CA, US, 2014a. DOI: https://doi.org/10.2514/6.2014-1774.
Laporte [2014b] Laporte, F., “JAC Software, solving conjunction assessment issues,” Proceedings of the Advanced Maui Optical and Space Surveillance Technologies Conference (AMOS), Maui, Hawaii, US, 2014b.
Tardioli and Vasile [2015] Tardioli, C., and Vasile, M., “Collision and re-entry analysis under aleatory and epistemic uncertainty,” Advances in Astronautical Sciences, Vol. 156, 2015, pp. 4205 – 4220.
Delande et al. [2018] Delande, E., Houssineau, J., and Jah, M., “A New Representation of uncertainty for data fusion in SSA Detection and Tracking Problems,” 2018 21st International Conference on Information Fusion (FUSION), Cambridge, United Kingdom, 2018. DOI: https://doi.org/10.23919/ICIF.2018.8455540.
Balch et al. [2019] Balch, M., Martin, R., and Ferson, S., “Satellite conjunction analysis and the false confidence theorem,” Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 475, No. 20180565, 2019. DOI: https://doi.org/10.1098/rspa.2018.0565.
Greco et al. [2021] Greco, C., Sánchez, L., and Vasile, M., “A robust Bayesian agent for optimal collision avoidance manoeuvre planning,” 8 ${}^{\text{th}}$ European Conference on Space Debris, ESA/ESOC, Darmstadt, Germany, 2021.
Sánchez and Vasile [2021] Sánchez, L., and Vasile, M., “On the use of machine learning and evidence theory to improve collision risk management,” Acta Astronautica, Vol. 181, 2021, pp. 694–706. DOI: https://doi.org/10.1016/j.actaastro.2020.08.004.
Sánchez and Vasile [2022] Sánchez, L., and Vasile, M., “Intelligent agent for decision-making support and collision avoidance manoeuvre design on space traffic management,” Advances in Space Research. In press, 2022. DOI: https://doi.org/10.1016/j.asr.2022.09.023.
Sánchez et al. [2022] Sánchez, L., Stevenson, E., Vasile, M., Rodríguez-Fernández, V., and Camacho, D., “An intelligent system for robust decision-making in the all-vs-all conjunction screening problem,” 3 ${}^{\text{rd}}$ IAA Conference on Space Situational Awareness (ICSSA), Tres Cantos, Madrid, Spain, 2022.
Helton et al. [2005] Helton, J. C., Oberkampf, W. L., and Johnson, J. D., “Competing failure risk analysis using evidence theory,” Risk Analysis, Vol. 25, No. 4, 2005, pp. 973–995. DOI: https://doi.org/10.1111/j.1539-6924.2005.00644.x.
Dvoretzky et al. [1956] Dvoretzky, A., Kiefer, J., and Wolfowitz, J., “Asymptotic minimax character of the sample distribution function and of the classical multinomial estimator,” Annals of Mathematical Statistics, Vol. 27, No. 3, 1956, pp. 642–669. DOI: https://doi.org/10.1214/aoms/1177728174.
Shafer [1976] Shafer, G., A Mathematical theory of evidence, 1^st ed., Princeton University Press, Princeton, NJ, 1976. ISBN: 9780691100425.
Serra et al. [2016] Serra, R., Arzelier, D., Joldes, M., Lasserre, J. B., Rondepierre, A., and Salvy, B., “Fast and accurate computation of orbital collision probability for short-term encounters,” Journal of Guidance, Control, and Dynamics, Vol. 39, 2016, pp. 1–13. DOI: https://doi.org/10.2514/1.G001353.
Greco and Vasile [2021] Greco, C., and Vasile, M., “Robust Bayesian particle filter for space object tracking under severe uncertainty,” Journal of Guidance, Control, and Dynamics, Vol. 45, No. 3, 2021, pp. 481–498. DOI: https://doi.org/10.2514/1.G006157.
Ferson et al. [2023] Ferson, S., Kreinovich, V., Ginzburg, L., Sentz, K., and Myers, D. S., “Constructing probability boxes and Dempster-Shafer structures,” Tech. rep., Sandia National Lab., Albuquerque, NM, United States, 2023. DOI: https://doi.org/10.2172/809606.
Ferson et al. [2007] Ferson, S., Kreinovich, V., Hajagos, J., Oberkampf, W., and Ginzburg, L., Experimental uncertainty estimation and statistics for data having interval uncertainty, Princeton University Press, Sandia National Laboratories (SNL), Albuquerque, NM, and Livermore, CA (United States), 2007, Chap. 4: Descriptive statistics for interval data, pp. 28–87. DOI: https://doi.org/10.2172/910198.
He et al. [2017] He, Y., Mirzargar, M., and Kirby, R. M., “An efficient reliability analysis approach for structure based on probability and probability box models,” Structural and Multidisciplinary Optimization, Vol. 56, 2017, pp. 167–181. DOI: https://doi.org/10.1007/s00158-017-1659-7.
ESA [2019] ESA, “European Space Agency: Kelvins collision avoidance challenge,” https://kelvins.esa.int/collision-avoidance-challenge/home/, 2019.
He et al. [2015] He, Y., Mirzargar, M., and Kirby, R. M., “Mixed aleatory and epistemic uncertainty quantification using fuzzy set theory,” International Journal of Approximate Reasoning, Vol. 66, 2015, pp. 1–15. DOI: https://doi.org/10.1016/j.ijar.2015.07.002.
Chojnacki et al. [2007] Chojnacki, E., Baccou, J., and Destercke, D., “Numerical sensitivity and efficiency in the treatment of epistemic and aleatory uncertainty,” 5 ${}^{th}$ International Conference on Sensitivity Analysis of Model Output, Budapest, Hungary, 2007. DOI: https://doi.org/10.1016/j.ijar.2015.07.002.
Ferson et al. [2004] Ferson, S., Nelsen, R. B., Hajagos, J., Berleant, D. J., Zhang, J., Tucker, W. T., Ginzburg, L. R., and Oberkampf, W. L., “Dependence in probabilistic modeling, Dempster-Shafer theory, and probability bounds analysis,” Tech. rep., Sandia National Lab., United States, Oct. 2004. DOI: https://doi.org/10.2172/919189.
Stroe et al. [2021] Stroe, I. F., Stanculescu, A. D., Ilioaica, P. B., Blaj, C. F., Nita, M. A., Butu, A. F., Escobar, D., Tirado, J., Bija, B., and Saez, D., “AUTOCA autonomous collision avoidance system,” 8 ${}^{\text{th}}$ European Conference on Space Debris, ESA/ESOC, Darmstadt, Germany, 2021.

Treatment of Epistemic Uncertainty in Conjunction Analysis with Dempster-Shafer Theory

Abstract

Abstract

1 Introduction

2 Conjunction Analysis with Dempster-Shafer Structures

3 Modelling Epistemic Uncertainty in Conjunction Data Messages

3.1 Scaling of the CDMs

3.2 α𝛼\alphaitalic_α-cuts and DSt Structures

4 Numerical experiments

4.1 Parameter Tuning

4.2 Comparison Against SDO and CNES

4.2.1 Space Debris Office Conjunction Risk Assessment

Event #1

Event #2

Event #3

4.2.2 CNES Conjunction Risk Assessment

Event #4

4.3 Statistical Analysis of CAM Executions

5 Conclusions

Acknowledgements

References

3.2 $\alpha$ -cuts and DSt Structures