The paper presents the results of a comparative evaluation of the predictive ability of seventeen spallation reaction models (CEM02, CEM03, Phits/jam, Cascade/ASF, Phits/Bertini, Bertini/Dresner, Cascade-4, INCL4/Abla, INCL4/smm, geant4/binary, Isabela/smm, geant4/Bertini, Isabela/Abla, INCL4/Gemini, CASCADeX-1.2, Isabel/Gemini, Phits/jqmd) for the interaction reactions of high-energy protons with ^natPb nuclei using the most popular methods of multiple-criteria decision analysis (MAVT/MAUT, AHP, TOPSIS, PROMETHEE). Multiple-criteria decision analysis methods are used extensively to support decision-making in various fields of knowledge, including nuclear physics and engineering, when aggregating conflicting criteria with due account for the expert and decision-maker opinions. Four factors of computational and experimental agreement (R, D, F, H), most commonly used in this field of knowledge, have been employed as the criteria, which, having been aggregated as part of applying respective multiple-criteria decision analysis methods, make it possible to estimate the integral measure of the computational model effectiveness and to rank the models, using this as the basis, depending on the degree of their predictive ability. It has been demonstrated that the ranking results obtained using different multiple-criteria decision analysis methods show a good agreement. Using a stochastic approach to the generation of weights, the models were ranked in conditions with the absence of data on the significance of individual agreement factors. Recommendations are presented for using the multiple-criteria decision analysis methods to address tasks involved in the preparation of nuclear data in conditions of a multiple-factor evaluation of discrepancies between calculations and experiment.

Нigh-energy nuclear reactions, nuclear data, multiple-criteria decision analysis methods, uncertainty

The tasks involved in design of high-energy neutron sources, production of medical isotopes, and protection against high-energy radiation of space vehicles and accelerators require a large number of nuclear data in a broad range of energies reaching tens of gigaelectronvolts. It is not possible to obtain all data experimentally due to which analytical methods are developed, the accuracy of these being checked by comparison with full-scale measurement data (Konobeyev et al. 2004, Leray 2009, Hendricks 2006).

There are numerous programs which enable calculation of various nuclear reactions for different types of incident particles, energy ranges and mass numbers of target nuclei. Various criteria and estimation techniques have been proposed for the quantitative comparison of calculation results with experimental data. However, there is no universal theoretical model that provides for a satisfactory description of the entire spectrum of nuclear reactions of practical interest since there is no versatile procedure to evaluate the predictive ability of computational tools which is expected to lead to different conclusions as to the most representative computational model.

The paper presents results of a multiple-criteria comparative evaluation of the predictive ability of seventeen spallation reaction models (CEM02, CEM03, Phits/jam, Cascade/ASF, Phits/Bertini, Bertini/Dresner, Cascade-4, INCL4/Abla, INCL4/smm, geant4/binary, Isabela/smm, geant4/Bertini, Isabela/Abla, INCL4/Gemini, CASCADeX-1.2, Isabel/Gemini, Phits/jqmd) for the interaction reactions of high-energy protons with ^natPb nuclei. The multiple-criteria comparison was based on the most popular methods of multiple-criteria decision analysis (MAVT/MAUT, AHP, TOPSIS, PROMETHEE), as well as on stochastic methods of evaluating the effects of the factor weight uncertainties on results which enable the ranking of models in conditions of no data available concerning the significance of individual agreement factors.

Computer modeling is the only possible way to describe the mechanism of the nucleon interaction in a high-energy region. Vector and parallel computations, which have become widespread recently, offer extensive capabilities for modeling a large number of events occurring within a short period of time. Validated models are included in radiation transport codes which makes it possible to calculate the effects of the formed particle interaction with the substance. In this connection, active work is under way to standardize the codes and parameters they comprise. Two possibilities for solving this problem are discussed. The first solution consists in selection of parameters and program modules to obtain the required data. The second one suggests standardization and coordination of fundamental parameters. There is however a probability that calculations performed with such set of parameters may have a worse agreement with the experiment. Cumulative information on the improved transport codes to study the radiation-substance interaction and the particle-nuclei interaction generators, including their respective peculiarities, is presented in Table 1 (Hendricks 2006, Sato et al. 2013, Agostinelliae et al. 2003, Battistoni et al. 2015, Mokhov et al. 2004).

Table 1.Download as CSV 

Most common modern transport codes.

Transport Language	Intranuclear cascade (pre-equilibrium)	De-excitation	Incident particle	Upper energy limit
MCNPX2.7 MCNP6 Fortran 90	Bertini (MPM)	Dresner or ABLA	n, p	3.5 GeV
			π	2.5 GeV
	Isabel (MPM)		n, p	0.8 GeV
			π	1.0 GeV
			d, t, 3He, α	1.0 GeV/nucleon
	INCL4.2		n, p	~3 GeV
			π	~2.5 GeV
			d, t, 3He, α	~3 GeV/nucleon
	CEM03 + GEM		n, p	5 GeV
			π	2.5 GeV
PHITS2.64 Fortran 77	INCL4.6	GEM	n, p	3 GeV
			π	3 GeV
			d, t, 3He, α	3 GeV/nucleon
GEANT4 C++	Bertini intranuclear cascade (+pre-equilibrium)	Internal evaporation (or GEM), fission, Multiple fragmentation, Fermi decay model or AblaV3	n, p	10 GeV
			π	10 GeV
	Binary cascade (+pre-equilibrium)		n, p	10 GeV
			π	10 GeV
			d, t, 3He, α	~3 – 5 GeV/nucleon
	INCL++		n, p	~3 GeV
			π	~3 GeV
			d, t, 3He, α	~3 GeV/nucleon
FLUKA Fortran 77	PEANUT (GINC+pre-equilibrium)	Internal evaporation, Fission and Fermi decay model	n, p	5 GeV
			π	5 GeV
	r-QMD-2.4		d, t, 3He, α	5 GeV/nucleon
MARS Fortran 77	CEM03	GEM	n, p	5 GeV
			π	5 GeV
	LAQGSM	GEM	d, t, 3He, α	800 GeV/nucleon

The intranuclear cascade model based on Monte Carlo method, coupled with an evaporative de-excitation model used to calculate the yields and characteristics of all particles formed in spallation reactions, has become widespread. Occasionally, pre-equilibrium emission of particles is introduced between the two stages. The descriptions of the nucleon-nucleon interaction processes practically coincide in all codes. Major discrepancies are found in the yield criteria at the intranuclear cascade stage, as well as in the model description of the pre-equilibrium stage and the cluster emission and pion formation process.

The energy range, in which this set of models is applicable, is rather wide: from several dozen megaelectronvolt to several gigaelectronvolt. Some code have, e.g., the INCL4 cascade model coupled with the ABLA evaporation model (Mank et al. 2008) lacking the pre-equilibrium stage. Calculations based on the INCL4/ABLA, CEM03 or LAQGSM code (Boudard et al. 2002, Mashnik et al. 2008, Mashnik 2001) provide for a good fit with the experimental data in a broad range of incident particle energies and target nuclei mass numbers. However, none of the existing models is capable to reproduce the experimental data across the energy interval and for all target nuclei.

In a set of cascade models, the model developed in Dubna in the 1960s (Barashenkov and Toneyev 1972) holds a special place. In this case, the development of the intranuclear cascade is modeled in time. For the past 20 years, this model was evolved at Obninsk Institute for Nuclear Power Engineering (OINPE, currently the Obninsk Branch of NRNU MEPhI), and the CASCADE/INPE code was developed on its basis (Barashenkov et al. 1999) for the particle transport calculations. This model was combined with the statistic model describing the equilibrium emission of particles. The new code called CASCADEX (CASCADE eXtended) (Andrianov et al. 2011) is designed to model the interaction of incident particles and nuclei with a mass number of up to 240 atomic mass units with substance. The mass numbers of the target nuclei (А) vary in a range of two to 240 amu. The incident particle energies are up to 2 GeV/nucleon for target nuclei with a mass of less than 40 amu and up to 1 GeV/nucleon for the nuclei heavier than 40 amu.

In 2008, as part of the respective IAEA joint project to verify spallation reaction models, a conclusion was made by experts in high energy physics that the existing models of reactions need to be verified based on all of the available set of experimental data so that to determine the accuracy and reliability of data obtained using these in various mass and energy ranges. It is reasonable to conduct a quantitative comparison of calculation results with experimental data as part of a multiple-criteria paradigm (by calculating the entire set of the calculation-experiment agreement factors).

To compare the calculation results for models with experimental data, the following agreement factors are used at the present time: F-, H-, R-, D-factors (see Table 2) (Andrianov et al. 2011a, Andrianov et al. 2016). As a rule, a single-criterion paradigm is used to interpret the evaluation results as one criterion is identified and the presence of the others is ignored. This provides for an unambiguous method to select the best calculation model for different nuclei and energy ranges or parameters of models. It should be noted that different research teams prefer different agreement criteria, this leading to different results. Attempts were made in some studies to evaluate the entire set of factors the results of which were used as the basis for the expert evaluation for the best model selection. All agreement factors can be taken into account simultaneously as part of implementing a multiple-criteria paradigm of evaluation based on decision-making support methods using multiple criteria, which makes it possible to consider the entire set of agreement criteria as well (Andrianov et al. 2013, Andrianov et al. 2017).

Table 2.Download as CSV 

Agreement factors.

F-factor
H-factor
D-factor
R-factor

To demonstrate the applicability of the multiple-criteria paradigm for evaluating the predictive ability of spallation reaction models, reactions of the interaction of a ^natPb target with a high-energy proton were considered. The selection of this type of reactions is connected with the fact that there is a large set of experimental data for the ^natPb target since lead is viewed as the base material for a number of accelerator driven system designs. The experimental values were taken from the EXFOR databases, as well as from the databases used in Benchmark of Spallation Models, an IAEA project. Excitation functions for the ^natPb(p,²⁰⁷Bi) reactions calculated using various models are presented in Figure 1 as an example. Table 3 presents agreement factors for the ^natPb(p,x) reaction. To evaluate the agreement factors, 279 experimental values of the ^natPb recoil nuclei cross-sections were selected with the incident proton energy values being in a range of 70 to 2600 MeV.

Figure 1. 

Excitation functions for the ^natPb(p,²⁰⁷Bi) reaction calculated based on different models: 1 – Cascad/ASF; 2 – Cascad-4; 3 – CEM-02; 4 – CASCADeX-1.2; 5 – INCL45/Abla07; 6 – geant4/binary; 7 – Bertini/Dresner; 8 – CEM-03; 9 – geant4/Bertini; 10 – Isabela/Abla07; 11 – INCL4/Gemini++; 12 – INCL45/smm; 13 – Isabel/Gemini; 14 – Phits/Bertini.

Table 3.Download as CSV 

Values of the ^natPb(p, x) reaction agreement factors.

Models of high-energy reactions	Agreement factors
	H	D	R	F
Cascade-4	6.17	0.69	0.91	5.14
Cascade / ASF	4.62	0.49	0.91	2.57
CASCADeX-1.2	5.82	0.71	0.46	10.98
CEM02	4.84	0.51	1.05	2.44
CEM03	5.21	0.56	1.06	2.46
geant4 / Bertini	14.80	1.02	1.40	4.00
geant4 / binary	4.39	0.53	0.69	3.73
INCL45 / Abla07	9.61	0.81	1.51	2.04
INCL45 / Gemini	20.26	1.28	2.04	2.48
INCL45 / smm	9.57	0.87	1.27	3.67
Bertini / Dresner	7.37	0.72	1.15	2.59
Isabela / Abla07	13.13	1.08	1.77	2.29
Isabel / Gemini	30.30	1.70	2.49	2.79
Isabela / smm	10.04	0.92	1.35	4.04
Phits / jqmd	42.86	2.23	2.26	6.43
Phits4 / jam	5.63	0.54	0.93	2.12
Phits / Bertini	6.75	0.61	1.16	2.08

Multiple-Criteria Decision Analysis (MCDA) methods are a tool designed to support decision-making by persons facing the necessity to make a choice in a situation characterized by multiple and contradictory factors (Yatsalo et al. 2016). These methods are intended to identify contradictions and to search for compromises in the process of decision-making. The problems for which the MCDA methods are designed consist of a finite number of alternatives each of which is represented by the quantitative evaluation of all of the criteria that characterize it and were defined explicitly at the beginning of the consideration process. A large number of the MCDA methods were developed for solving various problems (selection of the preferred alternative, ranking and screening). Each of the methods has its own advantages and drawbacks and can be more or less useful as the case may be.

To analyze the stability of the model ranking results with respect to the values of the factor weights that characterize the relative significance of comparison criteria, a stochastic approach was used to generate weights, this making it possible to evaluate the scatter in the final scores of models caused by the uncertainties of the weights and to rank models in conditions of no data available on the significance of individual agreement factors. It was assumed as part of this method that all of the weights had been distributed uniformly in a random manner in a range of zero to unity, with only the normalization condition (the total of the weights should be equal to unity in the framework of an additive MAVT model) superimposing on their potential values. The final scores for each of the considered models were evaluated based on MAVT for each set of weights. This makes it possible to determine the probability distribution functions for the final scores and rankings of models reflecting the influence of uncertainties in the factor weights. Based on this information, one can determine the probability of a particular model to be preferred. The ranking results can be shown as a ‘box-and-whisker’ diagram representing a convenient method to display numerical data broken down into four quartiles.

The estimates presented in this paper were obtained using the following well-known and broadly used MCDA methods, including MAVT (Multi-attribute Value Theory), MAUT (Multi-attribute Utility Theory), TOPSIS (Technique for Order Preference by Similarity to the Ideal Solution), PROMETHEE (Preference Ranking Organization Method for Enrichment Evaluations), and AHP (Analytic Hierarchy Process), which makes it possible to identify the robustness of the ranking results with respect to the ranking method used. All methods have been realized in their simplest form. It was assumed in the base calculation that all agreement factors are equally significant.

Table 4 shows the ranking results for models (ranks) obtained with the use of various methods and their respective groups. As can be seen, using various multiple-criteria decision analysis methods to evaluate the predictive ability of a spallation reaction leads, despite certain differences in the model ranking, to well-agreed and similar results. Despite the fact that the model ranking results are not affected by the weights of individual criteria, there are intervals in which the ranking procedures are preserved within broad variation limits of the weight values.

Table 4.Download as CSV 

Model ranking results for the ^natPb(p,x) reaction (equal weights).

Rank	MCDA methods				Model attractiveness group
	MAVT/ MAUT	AHP	TOPSIS	PROMETHEE
1	CEM02	CEM02	Phits4/jam	CEM02	1
2	Phits4/jam	Phits4/jam	CEM03	CEM03
3	Cascade/ASF	CEM03	Phits/Bertini	Phits4/jam
4	CEM03	Cascade/ASF	Cascade/ASF	Cascade/ASF
5	Phits/Bertini	Phits/Bertini	CEM02	Phits/Bertini
6	Bertini/Dresner	Bertini/Dresner	Phits/jqmd	Bertini/Dresner	2
7	Cascade 4	Cascade 4	Isabela/smm	Cascade 4
8	INCL45/abla07	INCL4/abla07	Cascade 4	INCL45/smm
9	INCL45/smm	Isabela/smm	INCL45/Abla	Isabela/smm
10	geant4/ binary	geant4/binary	geant4/ binary	geant4/binary
11	Isabela/smm	INCL4/smm	INCL45/Gemini	geant4/Bertini
12	geant4/Bertini	geant4/Bertin	Bertini/Dresner	INCL4/Abla07
13	Isabela/Abla07	Isabela/Abla07	geant4/Bertini	geant4/Bertini	3
14	INCL45/Gemini	INCL45/Gemini	Isabel/Gemini	INCL45/Gemini
15	CASCADeX-1.2	CASCADeX-1.2	INCL45/smm	CASCADeX-1.2
16	Isabel/Gemini	Isabel/Gemini	Isabela/Abla07	Isabel/Gemini
17	Phits/jqmd	Phits/jqmd	CASCADeX-1.2	Phits/jqmd

To update the values of the weights reflecting the expert representations concerning the importance of particular agreement factors, an expert evaluation is required to select their values. However, so that not to determine the values of weightings, one can evaluate the influence of the uncertainties in the weights on the final scores of the models by using the stochastic weight generation method which makes it possible to rank models in the absence of information on the significance of the agreement factors, as well as where it is required and probable that a particular model is preferred.

Figure 2 shows the MAVT model ranking results with regard for the uncertainties in the values of the weights in the box plot format (inverse distributions of 95, 75, 50, 25, and 5% are shown in the diagram). The models in the diagram are arranged in accordance with the average score values. An analysis of the uncertainty influence confirms the ranking results obtained using various methods. The best models are models of group 1, including CEM02, CEM03, Phits/jam, Cascade/ASF, Phits/Bertini. The Bertini/Dresner, Cascade-4, INCL4/Abla, INCL4/smm, geant4/binary, Isabela/smm, and geant4/Bertini models can be classified as models of attractiveness group 2. The Isabela/Abla, INCL4/Gemini, CASCADeX-1.2, Isabel/Gemini, and Phits/jqmd models are characterized by a great uncertainty and form attractiveness group 3.

Figure 2. 

Model ranking results with regard for the uncertainties in the weights of the agreement factors

When analyzing the obtained results, it is necessary to note that the CEM02, CEM03, Cascade/ASF, geant4/Bertini, and geant4/binary models, which do not contain a pre-equilibrium stage in their algorithm, belong to groups 1 and 3, which indicates that the advantages of taking into account the pre-equilibrium model are dubious. A major discrepancy in evaluating the predictive ability of the CASCADeX-1.2 code can be explained by the fact that the model built in it uses the Weisskopf-Ewing model (Weisskopf and Ewing 1940) instead of the commonly used Hauser-Feshbach formalism (Hauser and Feshbach 1952) to describe the slow-rate evaporation stage. For the time being, the model based on quantum-molecular dynamics (Phits/jqmd), despite a more complex representation of the reaction’s fast-cascade stage, describes inadequately the spallation reactions.

A multiple-criteria approach to evaluating the predictive abilities of high-energy nuclear reaction models based on multiple-criteria decision analysis methods provides for a more thorough differentiation among various models which serves an additional tool both for the understanding of the nuclear reaction mechanisms and for preparing a reliable array of nuclear data. The use of different multiple-criteria decision analysis methods for evaluating the predictive abilities of spallation reaction models shows that, despite certain differences in the model rankings, the results obtained using various methods prove to agree well. The results of the model ranking in conditions of uncertainties in the factor weights correlate with the ranking results obtained based on classical approaches. Based on the sensitivity analysis results, with regard for the additional analysis of alternatives using expert judgments and the entire set of graphic and attributive data, models of the CEM, Phits, and Cascade families can be regarded to be the best models.