Corresponding author: Andrey A. Andrianov (andreyandrianov@yandex.ru)
Academic editor: Yury Kazans
The paper presents the results of a comparative evaluation of the predictive ability of seventeen spallation reaction models (CEM02, CEM03, Phits/jam, Cascade/ASF, Phits/Bertini, Bertini/Dresner, Cascade4, INCL4/Abla, INCL4/smm, geant4/binary, Isabela/smm, geant4/Bertini, Isabela/Abla, INCL4/Gemini, CASCADeX1.2, Isabel/Gemini, Phits/jqmd) for the interaction reactions of highenergy protons with ^{nat}Pb nuclei using the most popular methods of multiplecriteria decision analysis (MAVT/MAUT, AHP, TOPSIS, PROMETHEE). Multiplecriteria decision analysis methods are used extensively to support decisionmaking in various fields of knowledge, including nuclear physics and engineering, when aggregating conflicting criteria with due account for the expert and decisionmaker opinions. Four factors of computational and experimental agreement (
The tasks involved in design of highenergy neutron sources, production of medical isotopes, and protection against highenergy radiation of space vehicles and accelerators require a large number of nuclear data in a broad range of energies reaching tens of gigaelectronvolts. It is not possible to obtain all data experimentally due to which analytical methods are developed, the accuracy of these being checked by comparison with fullscale measurement data (
There are numerous programs which enable calculation of various nuclear reactions for different types of incident particles, energy ranges and mass numbers of target nuclei. Various criteria and estimation techniques have been proposed for the quantitative comparison of calculation results with experimental data. However, there is no universal theoretical model that provides for a satisfactory description of the entire spectrum of nuclear reactions of practical interest since there is no versatile procedure to evaluate the predictive ability of computational tools which is expected to lead to different conclusions as to the most representative computational model.
The paper presents results of a multiplecriteria comparative evaluation of the predictive ability of seventeen spallation reaction models (CEM02, CEM03, Phits/jam, Cascade/ASF, Phits/Bertini, Bertini/Dresner, Cascade4, INCL4/Abla, INCL4/smm, geant4/binary, Isabela/smm, geant4/Bertini, Isabela/Abla, INCL4/Gemini, CASCADeX1.2, Isabel/Gemini, Phits/jqmd) for the interaction reactions of highenergy protons with ^{nat}Pb nuclei. The multiplecriteria comparison was based on the most popular methods of multiplecriteria decision analysis (MAVT/MAUT, AHP, TOPSIS, PROMETHEE), as well as on stochastic methods of evaluating the effects of the factor weight uncertainties on results which enable the ranking of models in conditions of no data available concerning the significance of individual agreement factors.
Computer modeling is the only possible way to describe the mechanism of the nucleon interaction in a highenergy region. Vector and parallel computations, which have become widespread recently, offer extensive capabilities for modeling a large number of events occurring within a short period of time. Validated models are included in radiation transport codes which makes it possible to calculate the effects of the formed particle interaction with the substance. In this connection, active work is under way to standardize the codes and parameters they comprise. Two possibilities for solving this problem are discussed. The first solution consists in selection of parameters and program modules to obtain the required data. The second one suggests standardization and coordination of fundamental parameters. There is however a probability that calculations performed with such set of parameters may have a worse agreement with the experiment. Cumulative information on the improved transport codes to study the radiationsubstance interaction and the particlenuclei interaction generators, including their respective peculiarities, is presented in Table
Most common modern transport codes.
Transport Language  Intranuclear cascade (preequilibrium)  Deexcitation  Incident particle  Upper energy limit 

MCNPX2.7 MCNP6 Fortran 90  Bertini (MPM)  Dresner or ABLA  n, p  3.5 GeV 
π  2.5 GeV  
Isabel (MPM)  n, p  0.8 GeV  
π  1.0 GeV  
d, t, ^{3}He, α  1.0 GeV/nucleon  
INCL4.2  n, p  ~3 GeV  
π  ~2.5 GeV  
d, t, ^{3}He, α  ~3 GeV/nucleon  
CEM03 + GEM  n, p  5 GeV  
π  2.5 GeV  
PHITS2.64 Fortran 77  INCL4.6  GEM  n, p  3 GeV 
π  3 GeV  
d, t, ^{3}He, α  3 GeV/nucleon  
GEANT4 C++  Bertini intranuclear cascade (+preequilibrium)  Internal evaporation (or GEM), fission, Multiple fragmentation, Fermi decay model or AblaV3  n, p  10 GeV 
π  10 GeV  
Binary cascade (+preequilibrium)  n, p  10 GeV  
π  10 GeV  
d, t, ^{3}He, α  ~3 – 5 GeV/nucleon  
INCL++  n, p  ~3 GeV  
π  ~3 GeV  
d, t, ^{3}He, α  ~3 GeV/nucleon  
FLUKA Fortran 77  PEANUT (GINC+preequilibrium)  Internal evaporation, Fission and Fermi decay model  n, p  5 GeV 
π  5 GeV  
rQMD2.4  d, t, ^{3}He, α  5 GeV/nucleon  
MARS Fortran 77  CEM03  GEM  n, p  5 GeV 
π  5 GeV  
LAQGSM  GEM  d, t, ^{3}He, α  800 GeV/nucleon 
The intranuclear cascade model based on Monte Carlo method, coupled with an evaporative deexcitation model used to calculate the yields and characteristics of all particles formed in spallation reactions, has become widespread. Occasionally, preequilibrium emission of particles is introduced between the two stages. The descriptions of the nucleonnucleon interaction processes practically coincide in all codes. Major discrepancies are found in the yield criteria at the intranuclear cascade stage, as well as in the model description of the preequilibrium stage and the cluster emission and pion formation process.
The energy range, in which this set of models is applicable, is rather wide: from several dozen megaelectronvolt to several gigaelectronvolt. Some code have, e.g., the INCL4 cascade model coupled with the ABLA evaporation model (
In a set of cascade models, the model developed in Dubna in the 1960s (
In 2008, as part of the respective IAEA joint project to verify spallation reaction models, a conclusion was made by experts in high energy physics that the existing models of reactions need to be verified based on all of the available set of experimental data so that to determine the accuracy and reliability of data obtained using these in various mass and energy ranges. It is reasonable to conduct a quantitative comparison of calculation results with experimental data as part of a multiplecriteria paradigm (by calculating the entire set of the calculationexperiment agreement factors).
To compare the calculation results for models with experimental data, the following agreement factors are used at the present time:
Agreement factors.








To demonstrate the applicability of the multiplecriteria paradigm for evaluating the predictive ability of spallation reaction models, reactions of the interaction of a ^{nat}Pb target with a highenergy proton were considered. The selection of this type of reactions is connected with the fact that there is a large set of experimental data for the ^{nat}Pb target since lead is viewed as the base material for a number of accelerator driven system designs. The experimental values were taken from the EXFOR databases, as well as from the databases used in Benchmark of Spallation Models, an IAEA project. Excitation functions for the ^{nat}Pb(p,^{207}Bi) reactions calculated using various models are presented in Figure
Excitation functions for the ^{nat}Pb(p,^{207}Bi) reaction calculated based on different models: 1 – Cascad/ASF; 2 – Cascad4; 3 – CEM02; 4 – CASCADeX1.2; 5 – INCL45/Abla07; 6 – geant4/binary; 7 – Bertini/Dresner; 8 – CEM03; 9 – geant4/Bertini; 10 – Isabela/Abla07; 11 – INCL4/Gemini++; 12 – INCL45/smm; 13 – Isabel/Gemini; 14 – Phits/Bertini.
Values of the ^{nat}Pb(p, x) reaction agreement factors.
Models of highenergy reactions  Agreement factors  






Cascade4  6.17  0.69  0.91  5.14 
Cascade / ASF  4.62  0.49  0.91  2.57 
CASCADeX1.2  5.82  0.71  0.46  10.98 
CEM02  4.84  0.51  1.05  2.44 
CEM03  5.21  0.56  1.06  2.46 
geant4 / Bertini  14.80  1.02  1.40  4.00 
geant4 / binary  4.39  0.53  0.69  3.73 
INCL45 / Abla07  9.61  0.81  1.51  2.04 
INCL45 / Gemini  20.26  1.28  2.04  2.48 
INCL45 / smm  9.57  0.87  1.27  3.67 
Bertini / Dresner  7.37  0.72  1.15  2.59 
Isabela / Abla07  13.13  1.08  1.77  2.29 
Isabel / Gemini  30.30  1.70  2.49  2.79 
Isabela / smm  10.04  0.92  1.35  4.04 
Phits / jqmd  42.86  2.23  2.26  6.43 
Phits4 / jam  5.63  0.54  0.93  2.12 
Phits / Bertini  6.75  0.61  1.16  2.08 
MultipleCriteria Decision Analysis (MCDA) methods are a tool designed to support decisionmaking by persons facing the necessity to make a choice in a situation characterized by multiple and contradictory factors (
To analyze the stability of the model ranking results with respect to the values of the factor weights that characterize the relative significance of comparison criteria, a stochastic approach was used to generate weights, this making it possible to evaluate the scatter in the final scores of models caused by the uncertainties of the weights and to rank models in conditions of no data available on the significance of individual agreement factors. It was assumed as part of this method that all of the weights had been distributed uniformly in a random manner in a range of zero to unity, with only the normalization condition (the total of the weights should be equal to unity in the framework of an additive MAVT model) superimposing on their potential values. The final scores for each of the considered models were evaluated based on MAVT for each set of weights. This makes it possible to determine the probability distribution functions for the final scores and rankings of models reflecting the influence of uncertainties in the factor weights. Based on this information, one can determine the probability of a particular model to be preferred. The ranking results can be shown as a ‘boxandwhisker’ diagram representing a convenient method to display numerical data broken down into four quartiles.
The estimates presented in this paper were obtained using the following wellknown and broadly used MCDA methods, including MAVT (Multiattribute Value Theory), MAUT (Multiattribute Utility Theory), TOPSIS (Technique for Order Preference by Similarity to the Ideal Solution), PROMETHEE (Preference Ranking Organization Method for Enrichment Evaluations), and AHP (Analytic Hierarchy Process), which makes it possible to identify the robustness of the ranking results with respect to the ranking method used. All methods have been realized in their simplest form. It was assumed in the base calculation that all agreement factors are equally significant.
Table
Model ranking results for the ^{nat}Pb(p,x) reaction (equal weights).
Rank  MCDA methods  Model attractiveness group  

MAVT/ MAUT  AHP  TOPSIS  PROMETHEE  
1  CEM02  CEM02  Phits4/jam  CEM02  1 
2  Phits4/jam  Phits4/jam  CEM03  CEM03  
3  Cascade/ASF  CEM03  Phits/Bertini  Phits4/jam  
4  CEM03  Cascade/ASF  Cascade/ASF  Cascade/ASF  
5  Phits/Bertini  Phits/Bertini  CEM02  Phits/Bertini  
6  Bertini/Dresner  Bertini/Dresner  Phits/jqmd  Bertini/Dresner  2 
7  Cascade 4  Cascade 4  Isabela/smm  Cascade 4  
8  INCL45/abla07  INCL4/abla07  Cascade 4  INCL45/smm  
9  INCL45/smm  Isabela/smm  INCL45/Abla  Isabela/smm  
10  geant4/ binary  geant4/binary  geant4/ binary  geant4/binary  
11  Isabela/smm  INCL4/smm  INCL45/Gemini  geant4/Bertini  
12  geant4/Bertini  geant4/Bertin  Bertini/Dresner  INCL4/Abla07  
13  Isabela/Abla07  Isabela/Abla07  geant4/Bertini  geant4/Bertini  3 
14  INCL45/Gemini  INCL45/Gemini  Isabel/Gemini  INCL45/Gemini  
15  CASCADeX1.2  CASCADeX1.2  INCL45/smm  CASCADeX1.2  
16  Isabel/Gemini  Isabel/Gemini  Isabela/Abla07  Isabel/Gemini  
17  Phits/jqmd  Phits/jqmd  CASCADeX1.2  Phits/jqmd 
To update the values of the weights reflecting the expert representations concerning the importance of particular agreement factors, an expert evaluation is required to select their values. However, so that not to determine the values of weightings, one can evaluate the influence of the uncertainties in the weights on the final scores of the models by using the stochastic weight generation method which makes it possible to rank models in the absence of information on the significance of the agreement factors, as well as where it is required and probable that a particular model is preferred.
Figure
Model ranking results with regard for the uncertainties in the weights of the agreement factors
When analyzing the obtained results, it is necessary to note that the CEM02, CEM03, Cascade/ASF, geant4/Bertini, and geant4/binary models, which do not contain a preequilibrium stage in their algorithm, belong to groups 1 and 3, which indicates that the advantages of taking into account the preequilibrium model are dubious. A major discrepancy in evaluating the predictive ability of the CASCADeX1.2 code can be explained by the fact that the model built in it uses the WeisskopfEwing model (
A multiplecriteria approach to evaluating the predictive abilities of highenergy nuclear reaction models based on multiplecriteria decision analysis methods provides for a more thorough differentiation among various models which serves an additional tool both for the understanding of the nuclear reaction mechanisms and for preparing a reliable array of nuclear data. The use of different multiplecriteria decision analysis methods for evaluating the predictive abilities of spallation reaction models shows that, despite certain differences in the model rankings, the results obtained using various methods prove to agree well. The results of the model ranking in conditions of uncertainties in the factor weights correlate with the ranking results obtained based on classical approaches. Based on the sensitivity analysis results, with regard for the additional analysis of alternatives using expert judgments and the entire set of graphic and attributive data, models of the CEM, Phits, and Cascade families can be regarded to be the best models.
✩ Russian text published: Izvestiya vuzov. Yadernaya Energetika (ISSN 02043327), 2018, n. 2, pp. 157–168.