Comparison of spallation reaction models based on multiple-criteria decision analysis
expand article infoAndrey A. Andrianov, Yury A. Korovin, Ilya S. Kuptsov, Aleksandr Yu. Konobeyev§, Olga N. Andrianova|
‡ NRNU MEPhI, Moscow, Russia
§ Karlsruhe Institute of Technology, Eggenstein-Leopoldshafen, Germany
| JSC “SSC RF-IPPE n.a. A.I. Leypunsky", Obninsk, Russia
Open Access


The paper presents the results of a comparative evaluation of the predictive ability of seventeen spallation reaction models (CEM02, CEM03, Phits/jam, Cascade/ASF, Phits/Bertini, Bertini/Dresner, Cascade-4, INCL4/Abla, INCL4/smm, geant4/binary, Isabela/smm, geant4/Bertini, Isabela/Abla, INCL4/Gemini, CASCADeX-1.2, Isabel/Gemini, Phits/jqmd) for the interaction reactions of high-energy protons with natPb nuclei using the most popular methods of multiple-criteria decision analysis (MAVT/MAUT, AHP, TOPSIS, PROMETHEE). Multiple-criteria decision analysis methods are used extensively to support decision-making in various fields of knowledge, including nuclear physics and engineering, when aggregating conflicting criteria with due account for the expert and decision-maker opinions. Four factors of computational and experimental agreement (R, D, F, H), most commonly used in this field of knowledge, have been employed as the criteria, which, having been aggregated as part of applying respective multiple-criteria decision analysis methods, make it possible to estimate the integral measure of the computational model effectiveness and to rank the models, using this as the basis, depending on the degree of their predictive ability. It has been demonstrated that the ranking results obtained using different multiple-criteria decision analysis methods show a good agreement. Using a stochastic approach to the generation of weights, the models were ranked in conditions with the absence of data on the significance of individual agreement factors. Recommendations are presented for using the multiple-criteria decision analysis methods to address tasks involved in the preparation of nuclear data in conditions of a multiple-factor evaluation of discrepancies between calculations and experiment.


Нigh-energy nuclear reactions, nuclear data, multiple-criteria decision analysis methods, uncertainty


The tasks involved in design of high-energy neutron sources, production of medical isotopes, and protection against high-energy radiation of space vehicles and accelerators require a large number of nuclear data in a broad range of energies reaching tens of gigaelectronvolts. It is not possible to obtain all data experimentally due to which analytical methods are developed, the accuracy of these being checked by comparison with full-scale measurement data (Konobeyev et al. 2004, Leray 2009, Hendricks 2006).

There are numerous programs which enable calculation of various nuclear reactions for different types of incident particles, energy ranges and mass numbers of target nuclei. Various criteria and estimation techniques have been proposed for the quantitative comparison of calculation results with experimental data. However, there is no universal theoretical model that provides for a satisfactory description of the entire spectrum of nuclear reactions of practical interest since there is no versatile procedure to evaluate the predictive ability of computational tools which is expected to lead to different conclusions as to the most representative computational model.

The paper presents results of a multiple-criteria comparative evaluation of the predictive ability of seventeen spallation reaction models (CEM02, CEM03, Phits/jam, Cascade/ASF, Phits/Bertini, Bertini/Dresner, Cascade-4, INCL4/Abla, INCL4/smm, geant4/binary, Isabela/smm, geant4/Bertini, Isabela/Abla, INCL4/Gemini, CASCADeX-1.2, Isabel/Gemini, Phits/jqmd) for the interaction reactions of high-energy protons with natPb nuclei. The multiple-criteria comparison was based on the most popular methods of multiple-criteria decision analysis (MAVT/MAUT, AHP, TOPSIS, PROMETHEE), as well as on stochastic methods of evaluating the effects of the factor weight uncertainties on results which enable the ranking of models in conditions of no data available concerning the significance of individual agreement factors.

Modern spallation reaction models

Computer modeling is the only possible way to describe the mechanism of the nucleon interaction in a high-energy region. Vector and parallel computations, which have become widespread recently, offer extensive capabilities for modeling a large number of events occurring within a short period of time. Validated models are included in radiation transport codes which makes it possible to calculate the effects of the formed particle interaction with the substance. In this connection, active work is under way to standardize the codes and parameters they comprise. Two possibilities for solving this problem are discussed. The first solution consists in selection of parameters and program modules to obtain the required data. The second one suggests standardization and coordination of fundamental parameters. There is however a probability that calculations performed with such set of parameters may have a worse agreement with the experiment. Cumulative information on the improved transport codes to study the radiation-substance interaction and the particle-nuclei interaction generators, including their respective peculiarities, is presented in Table 1 (Hendricks 2006, Sato et al. 2013, Agostinelliae et al. 2003, Battistoni et al. 2015, Mokhov et al. 2004).

Most common modern transport codes.

Transport Language Intranuclear cascade (pre-equilibrium) De-excitation Incident particle Upper energy limit
MCNPX2.7 MCNP6 Fortran 90 Bertini (MPM) Dresner or ABLA n, p 3.5 GeV
π 2.5 GeV
Isabel (MPM) n, p 0.8 GeV
π 1.0 GeV
d, t, 3He, α 1.0 GeV/nucleon
INCL4.2 n, p ~3 GeV
π ~2.5 GeV
d, t, 3He, α ~3 GeV/nucleon
CEM03 + GEM n, p 5 GeV
π 2.5 GeV
PHITS2.64 Fortran 77 INCL4.6 GEM n, p 3 GeV
π 3 GeV
d, t, 3He, α 3 GeV/nucleon
GEANT4 C++ Bertini intranuclear cascade (+pre-equilibrium) Internal evaporation (or GEM), fission, Multiple fragmentation, Fermi decay model or AblaV3 n, p 10 GeV
π 10 GeV
Binary cascade (+pre-equilibrium) n, p 10 GeV
π 10 GeV
d, t, 3He, α ~3 – 5 GeV/nucleon
INCL++ n, p ~3 GeV
π ~3 GeV
d, t, 3He, α ~3 GeV/nucleon
FLUKA Fortran 77 PEANUT (GINC+pre-equilibrium) Internal evaporation, Fission and Fermi decay model n, p 5 GeV
π 5 GeV
r-QMD-2.4 d, t, 3He, α 5 GeV/nucleon
MARS Fortran 77 CEM03 GEM n, p 5 GeV
π 5 GeV
LAQGSM GEM d, t, 3He, α 800 GeV/nucleon

The intranuclear cascade model based on Monte Carlo method, coupled with an evaporative de-excitation model used to calculate the yields and characteristics of all particles formed in spallation reactions, has become widespread. Occasionally, pre-equilibrium emission of particles is introduced between the two stages. The descriptions of the nucleon-nucleon interaction processes practically coincide in all codes. Major discrepancies are found in the yield criteria at the intranuclear cascade stage, as well as in the model description of the pre-equilibrium stage and the cluster emission and pion formation process.

The energy range, in which this set of models is applicable, is rather wide: from several dozen megaelectronvolt to several gigaelectronvolt. Some code have, e.g., the INCL4 cascade model coupled with the ABLA evaporation model (Mank et al. 2008) lacking the pre-equilibrium stage. Calculations based on the INCL4/ABLA, CEM03 or LAQGSM code (Boudard et al. 2002, Mashnik et al. 2008, Mashnik 2001) provide for a good fit with the experimental data in a broad range of incident particle energies and target nuclei mass numbers. However, none of the existing models is capable to reproduce the experimental data across the energy interval and for all target nuclei.

In a set of cascade models, the model developed in Dubna in the 1960s (Barashenkov and Toneyev 1972) holds a special place. In this case, the development of the intranuclear cascade is modeled in time. For the past 20 years, this model was evolved at Obninsk Institute for Nuclear Power Engineering (OINPE, currently the Obninsk Branch of NRNU MEPhI), and the CASCADE/INPE code was developed on its basis (Barashenkov et al. 1999) for the particle transport calculations. This model was combined with the statistic model describing the equilibrium emission of particles. The new code called CASCADEX (CASCADE eXtended) (Andrianov et al. 2011) is designed to model the interaction of incident particles and nuclei with a mass number of up to 240 atomic mass units with substance. The mass numbers of the target nuclei (А) vary in a range of two to 240 amu. The incident particle energies are up to 2 GeV/nucleon for target nuclei with a mass of less than 40 amu and up to 1 GeV/nucleon for the nuclei heavier than 40 amu.

In 2008, as part of the respective IAEA joint project to verify spallation reaction models, a conclusion was made by experts in high energy physics that the existing models of reactions need to be verified based on all of the available set of experimental data so that to determine the accuracy and reliability of data obtained using these in various mass and energy ranges. It is reasonable to conduct a quantitative comparison of calculation results with experimental data as part of a multiple-criteria paradigm (by calculating the entire set of the calculation-experiment agreement factors).

Agreement factors

To compare the calculation results for models with experimental data, the following agreement factors are used at the present time: F-, H-, R-, D-factors (see Table 2) (Andrianov et al. 2011a, Andrianov et al. 2016). As a rule, a single-criterion paradigm is used to interpret the evaluation results as one criterion is identified and the presence of the others is ignored. This provides for an unambiguous method to select the best calculation model for different nuclei and energy ranges or parameters of models. It should be noted that different research teams prefer different agreement criteria, this leading to different results. Attempts were made in some studies to evaluate the entire set of factors the results of which were used as the basis for the expert evaluation for the best model selection. All agreement factors can be taken into account simultaneously as part of implementing a multiple-criteria paradigm of evaluation based on decision-making support methods using multiple criteria, which makes it possible to consider the entire set of agreement criteria as well (Andrianov et al. 2013, Andrianov et al. 2017).

Agreement factors.





To demonstrate the applicability of the multiple-criteria paradigm for evaluating the predictive ability of spallation reaction models, reactions of the interaction of a natPb target with a high-energy proton were considered. The selection of this type of reactions is connected with the fact that there is a large set of experimental data for the natPb target since lead is viewed as the base material for a number of accelerator driven system designs. The experimental values were taken from the EXFOR databases, as well as from the databases used in Benchmark of Spallation Models, an IAEA project. Excitation functions for the natPb(p,207Bi) reactions calculated using various models are presented in Figure 1 as an example. Table 3 presents agreement factors for the natPb(p,x) reaction. To evaluate the agreement factors, 279 experimental values of the natPb recoil nuclei cross-sections were selected with the incident proton energy values being in a range of 70 to 2600 MeV.

Figure 1. 

Excitation functions for the natPb(p,207Bi) reaction calculated based on different models: 1 – Cascad/ASF; 2 – Cascad-4; 3 – CEM-02; 4 – CASCADeX-1.2; 5 – INCL45/Abla07; 6 – geant4/binary; 7 – Bertini/Dresner; 8 – CEM-03; 9 – geant4/Bertini; 10 – Isabela/Abla07; 11 – INCL4/Gemini++; 12 – INCL45/smm; 13 – Isabel/Gemini; 14 – Phits/Bertini.

Values of the natPb(p, x) reaction agreement factors.

Models of high-energy reactions Agreement factors
Cascade-4 6.17 0.69 0.91 5.14
Cascade / ASF 4.62 0.49 0.91 2.57
CASCADeX-1.2 5.82 0.71 0.46 10.98
CEM02 4.84 0.51 1.05 2.44
CEM03 5.21 0.56 1.06 2.46
geant4 / Bertini 14.80 1.02 1.40 4.00
geant4 / binary 4.39 0.53 0.69 3.73
INCL45 / Abla07 9.61 0.81 1.51 2.04
INCL45 / Gemini 20.26 1.28 2.04 2.48
INCL45 / smm 9.57 0.87 1.27 3.67
Bertini / Dresner 7.37 0.72 1.15 2.59
Isabela / Abla07 13.13 1.08 1.77 2.29
Isabel / Gemini 30.30 1.70 2.49 2.79
Isabela / smm 10.04 0.92 1.35 4.04
Phits / jqmd 42.86 2.23 2.26 6.43
Phits4 / jam 5.63 0.54 0.93 2.12
Phits / Bertini 6.75 0.61 1.16 2.08

Multiple-criteria decision analysis methods used

Multiple-Criteria Decision Analysis (MCDA) methods are a tool designed to support decision-making by persons facing the necessity to make a choice in a situation characterized by multiple and contradictory factors (Yatsalo et al. 2016). These methods are intended to identify contradictions and to search for compromises in the process of decision-making. The problems for which the MCDA methods are designed consist of a finite number of alternatives each of which is represented by the quantitative evaluation of all of the criteria that characterize it and were defined explicitly at the beginning of the consideration process. A large number of the MCDA methods were developed for solving various problems (selection of the preferred alternative, ranking and screening). Each of the methods has its own advantages and drawbacks and can be more or less useful as the case may be.

To analyze the stability of the model ranking results with respect to the values of the factor weights that characterize the relative significance of comparison criteria, a stochastic approach was used to generate weights, this making it possible to evaluate the scatter in the final scores of models caused by the uncertainties of the weights and to rank models in conditions of no data available on the significance of individual agreement factors. It was assumed as part of this method that all of the weights had been distributed uniformly in a random manner in a range of zero to unity, with only the normalization condition (the total of the weights should be equal to unity in the framework of an additive MAVT model) superimposing on their potential values. The final scores for each of the considered models were evaluated based on MAVT for each set of weights. This makes it possible to determine the probability distribution functions for the final scores and rankings of models reflecting the influence of uncertainties in the factor weights. Based on this information, one can determine the probability of a particular model to be preferred. The ranking results can be shown as a ‘box-and-whisker’ diagram representing a convenient method to display numerical data broken down into four quartiles.

Model ranking results

The estimates presented in this paper were obtained using the following well-known and broadly used MCDA methods, including MAVT (Multi-attribute Value Theory), MAUT (Multi-attribute Utility Theory), TOPSIS (Technique for Order Preference by Similarity to the Ideal Solution), PROMETHEE (Preference Ranking Organization Method for Enrichment Evaluations), and AHP (Analytic Hierarchy Process), which makes it possible to identify the robustness of the ranking results with respect to the ranking method used. All methods have been realized in their simplest form. It was assumed in the base calculation that all agreement factors are equally significant.

Table 4 shows the ranking results for models (ranks) obtained with the use of various methods and their respective groups. As can be seen, using various multiple-criteria decision analysis methods to evaluate the predictive ability of a spallation reaction leads, despite certain differences in the model ranking, to well-agreed and similar results. Despite the fact that the model ranking results are not affected by the weights of individual criteria, there are intervals in which the ranking procedures are preserved within broad variation limits of the weight values.

Model ranking results for the natPb(p,x) reaction (equal weights).

Rank MCDA methods Model attractiveness group
1 CEM02 CEM02 Phits4/jam CEM02 1
2 Phits4/jam Phits4/jam CEM03 CEM03
3 Cascade/ASF CEM03 Phits/Bertini Phits4/jam
4 CEM03 Cascade/ASF Cascade/ASF Cascade/ASF
5 Phits/Bertini Phits/Bertini CEM02 Phits/Bertini
6 Bertini/Dresner Bertini/Dresner Phits/jqmd Bertini/Dresner 2
7 Cascade 4 Cascade 4 Isabela/smm Cascade 4
8 INCL45/abla07 INCL4/abla07 Cascade 4 INCL45/smm
9 INCL45/smm Isabela/smm INCL45/Abla Isabela/smm
10 geant4/ binary geant4/binary geant4/ binary geant4/binary
11 Isabela/smm INCL4/smm INCL45/Gemini geant4/Bertini
12 geant4/Bertini geant4/Bertin Bertini/Dresner INCL4/Abla07
13 Isabela/Abla07 Isabela/Abla07 geant4/Bertini geant4/Bertini 3
14 INCL45/Gemini INCL45/Gemini Isabel/Gemini INCL45/Gemini
15 CASCADeX-1.2 CASCADeX-1.2 INCL45/smm CASCADeX-1.2
16 Isabel/Gemini Isabel/Gemini Isabela/Abla07 Isabel/Gemini
17 Phits/jqmd Phits/jqmd CASCADeX-1.2 Phits/jqmd

To update the values of the weights reflecting the expert representations concerning the importance of particular agreement factors, an expert evaluation is required to select their values. However, so that not to determine the values of weightings, one can evaluate the influence of the uncertainties in the weights on the final scores of the models by using the stochastic weight generation method which makes it possible to rank models in the absence of information on the significance of the agreement factors, as well as where it is required and probable that a particular model is preferred.

Figure 2 shows the MAVT model ranking results with regard for the uncertainties in the values of the weights in the box plot format (inverse distributions of 95, 75, 50, 25, and 5% are shown in the diagram). The models in the diagram are arranged in accordance with the average score values. An analysis of the uncertainty influence confirms the ranking results obtained using various methods. The best models are models of group 1, including CEM02, CEM03, Phits/jam, Cascade/ASF, Phits/Bertini. The Bertini/Dresner, Cascade-4, INCL4/Abla, INCL4/smm, geant4/binary, Isabela/smm, and geant4/Bertini models can be classified as models of attractiveness group 2. The Isabela/Abla, INCL4/Gemini, CASCADeX-1.2, Isabel/Gemini, and Phits/jqmd models are characterized by a great uncertainty and form attractiveness group 3.

Figure 2. 

Model ranking results with regard for the uncertainties in the weights of the agreement factors

When analyzing the obtained results, it is necessary to note that the CEM02, CEM03, Cascade/ASF, geant4/Bertini, and geant4/binary models, which do not contain a pre-equilibrium stage in their algorithm, belong to groups 1 and 3, which indicates that the advantages of taking into account the pre-equilibrium model are dubious. A major discrepancy in evaluating the predictive ability of the CASCADeX-1.2 code can be explained by the fact that the model built in it uses the Weisskopf-Ewing model (Weisskopf and Ewing 1940) instead of the commonly used Hauser-Feshbach formalism (Hauser and Feshbach 1952) to describe the slow-rate evaporation stage. For the time being, the model based on quantum-molecular dynamics (Phits/jqmd), despite a more complex representation of the reaction’s fast-cascade stage, describes inadequately the spallation reactions.


A multiple-criteria approach to evaluating the predictive abilities of high-energy nuclear reaction models based on multiple-criteria decision analysis methods provides for a more thorough differentiation among various models which serves an additional tool both for the understanding of the nuclear reaction mechanisms and for preparing a reliable array of nuclear data. The use of different multiple-criteria decision analysis methods for evaluating the predictive abilities of spallation reaction models shows that, despite certain differences in the model rankings, the results obtained using various methods prove to agree well. The results of the model ranking in conditions of uncertainties in the factor weights correlate with the ranking results obtained based on classical approaches. Based on the sensitivity analysis results, with regard for the additional analysis of alternatives using expert judgments and the entire set of graphic and attributive data, models of the CEM, Phits, and Cascade families can be regarded to be the best models.


  • Andrianov A, Dogov A, Kuptsov I, Svetlichny L, Korovin Yu (2016) Integrated software tools for radiation damage, activation and transmutation studies in advanced nuclear systems. In: Physics of Reactors (2016) Proceedings: PHYSOR 2016 Conference: Unifying Theory and Experiments in the 21st Century Unifying Theory and Experiments in the 21st Century, 3020–3029.
  • Andrianov A, Kuptsov I, Andrianova O, Konobeyev A, Korovin Yu (2017) Multi-criteria comparative evaluation of spallation reaction models. Proc.: EPJ Web of Conferences 22, 2017, Series: ND 2016: International Conference on Nuclear Data for Science and Technology: 12007.
  • Andrianov AA, Gritsyuk SV, Korovin YuA, Kuptsov IS (2013) Multi-criteria comparative evaluation of spallation reaction models. Vestnik natsionalnogo issledovatelskogo yadernogo universiteta MIFI, 2(2): 226. [In Russian]
  • Andrianov AA, Konobeyev AYu, Korovin YuA, Kuptsov IS, Stankovskiy AYu (2011) The improved CASCADeX 1.2 code for spallation reaction calculations. Izvestiya vysshykh uchebnykh zavedeniy. Yadernaya energetika, 2: 5–16. [In Russian]
  • Andrianov AA, Korovin YA, Kuptsov IS, Stankovskiy AY (2011a) Interactive information system for preparation and verification of nuclear data in the high-energy range. Journal of the Korean Physical Society, 59(23): 1096–1099.
  • Barashenkov VS, Konobeyev AYu, Korovin YuA, Sosnin VN (1999) CASCADE/INPE code system, Atomnaya energiya (Atomic Energy) 87(4): 742–744. [In Russian]
  • Barashenkov VS, Toneyev VD (1972) Interactions of high-energy particles and atomic nuclei with nuclei. Moscow. Atomizdat Publ., 351 pp. [In Russian]
  • Boudard A, Cugnon J, Leray S, Volant C (2002) Intranuclear cascade model for a comprehensive description of spallation reaction data. Phys. Rev. C, 66(044615): 1–28.
  • Hendricks JS (2006) MCNPX Version 26C. Report LA-UR-06-7991.
  • Konobeyev AYu, Korovin YuA, Pilnov GB, Stankovskiy AYu, Andrianov AA (2004) Evaluated Transport Files to Study the Particle Transport in Materials Irradiated by Neutrons with Energies up to 150 MeV. Izvestiya vysshykh uchebnykh zavedeniy. Yadernaya Energetika, 4: 56–62. [In Russian]
  • Leray S (2009) Needs for a benchmark of spallation models for reliable simulation of spallation related applications / PSI Proceedings 09–01. ARIA: 89.
  • Mashnik S (2001) Validation and Verification of MCNP6 Against High-Energy Experimental Data and Calculations by Other Codes. II. The LAQGSM Testing Primer. LANL Report LA-UR-11-05627.
  • Mashnik SG, Gudima KK, Prael RE, Sierk AJ, Baznat MI, Mokhov NV (2008) CEM03.03 and LAQGSM03.03 Event Generators for the MCNP6, MCNPX, and MARS15 Transport Codes. LANL Report LA-UR-08-2931.
  • Mokhov NV, Gudima KK, Mashnik SG, Kostin MA (2004) Physics Models in the MARS15 Code for Accelerator and Space Applications, Fermilab-Conf-04/269-AD, ND2004 paper.
  • Sato T, Niita K, Matsuda N, Hashimoto Sh, Iwamoto Y, Noda Sh, Ogawa T, Iwase H, Nakashima H, Fukahori T, Okumura K, Kai T, Chiba S, Furuta T, Sihver L (2013) Particle and Heavy Ion Transport Code System, PHITS, version 2.52. Journal of Nuclear Science and Technology, 50(9): 913–923.
  • Yatsalo B, Gritsyuk S, Sullivan T, Trump B, Linkov I (2016) Multi-criteria risk management with the use of DecernsMCDA: methods and case studies. Environment Systems and Decisions, 36(3): 266–276.

✩ Russian text published: Izvestiya vuzov. Yadernaya Energetika (ISSN 0204-3327), 2018, n. 2, pp. 157–168.