XMM links

General X-ray data reduction links

XSPEC error codes from tclout command page:
TFFFFFFFF = new minimum found
FTFFFFFFF = non-monotonicity detected
FFTFFFFFF = minimization may have run into problem (?)
FFFTFFFFF = hit hard lower limit
FFFFTFFFF = hit hard upper limit
FFFFFTFFF = parameter frozen
FFFFFFTFF = search failed in -ve direction
FFFFFFFTF = search failed in +ve direction
FFFFFFFFT = reduced chi-squared too high

Resources for XMM-Newton EPIC MOS/PN data reduction

Below are some things I ran into while reducing XMM imaging data (MOS full frame; PN full frame and large window) with XMM SAS v14/v15 on CentOS (xmmsas_20141104_1833, xmmsas_20160201_1833). I post my notes in case the information is of use to anyone out there working on XMM data, but as usual correctness is not guaranteed. Comments, questions, corrections, and diatribes are welcome; mail to forename.surname@berkeley.edu.

General comment: there are often multiple ways to specify the same command in SAS, which can make comparing command invocations a bit tricky. A number of parameters are not documented, and/or are automatically set when arguments are passed to a linked parameter.

SWCX/SP Tool (web form link), from the US XMM-Newton GOF, provides estimated time series of solar wind charge exchange (SWCX) emission by integrating scaled charge exchange emissivity along line-of-sight through the magnetosheath (words taken near verbatim from their site). Also observation light curves, solar wind abundances and proton densities, and wire plots of XMM's attitude and location within the magnetosheath.

Reduction setup / parameters

Recommendations from XMM-SOC-CAL-TN-0018, Sec. 11.2:

Reference on event patterns (same as ASCA/Chandra event grades):

e[p,m]chain vs. e[p,m]proc: expected to be the same, just script vs. compiled implementation (helpdesk thread reply 2)

Helpdesk thread on PN calibration uncertainties at low energies (0.1-0.3 keV, and stuff from 1 keV down): link; see A. Lia's suggested ref Tsujimoto et al. (2011A&A...525A..25T). We should be careful with PN data below 1 keV.

Loads of archived EPIC calibration presentations: link
Reference slides on soft proton flares by Kuntz et al., 2014. Useful plot of proton (flare) vignetting function, which is sharply peaked on-axis then quickly flattens to ~0.6 at 1 arcmin. out.

Key reference on MOS background: Kuntz & Snowden (2008). Some recent (2017) work on XMM background: intro (Molendi), data (Marelli+), MOS in-FOV (Salvetti+), location-dependence (Ghizzardi+), where is the noise coming from (Gastaldello+), and accompanying IACHEC presentation by Molendi.

Generalities on SAS, ESAS

epreject

Scribbled notes from a first pass of figuring out how epreject works, and how to assess the effect of its energy offset corrections.

The main issue I ran into: if you are running epreject through epchain or epproc, many intermediate steps are obscured. Early observations may not have available offset maps or reference offset maps (I'm not sure why the latter are missing from the CCF).

The suggestion to create a diagnostic image with PI between 120 and 200 was most useful. Events with ADU<20 are cut in epchain processing, and the image derived from intermediate products was hard to interpret (mostly just looked like a bunch of noise all over CCD 1).

I found it helpful to plot histograms in PHA and PI space, using the output event lists, and confirm that there was (1) a slight decrease in low energy events, and (2) a shift of a low energy feature upwards in channel/energy. But, other than that, the counts looked the same. At most, there might have been a shift in PI space out to ~2 keV, but the presence of a shift is most obvious for some peaks around 0.24 keV, 0.34 keV, 0.54 keV.

The user support guide section 4.3.2.1.1 (link) explains things in a little more detail than the epreject doc.

evselect

Conceptually, evselect operates by (1) filtering, and maybe keeping filtered output if you ask for it, and (2) creating some other output (histogram, light curve, image, etc.).

The behavior of evselect can be confusing. A few parameters are not documented (withfilteredset, destruct). And, setting a parameter sometimes alters another parameter's default value, unbeknownst to the user. One major consequence is that explicitly setting a parameter to its default value can still cause evselect to change other parameter values to "special case defaults". This allows evselect to create output that aligns with the typical user's expectations, but makes parameter interdependence a bit more complex.

Here's an example for the three parameters that control filtering output (two of which are officially documented): 

{none specified}   ---> keepfilteroutput=N, withfilteredset=Y, filteredset=filtered.fits  // Default, no filtering files output
keepfilteroutput=N ---> keepfilteroutput=N, withfilteredset=Y, filteredset=filtered.fits  // Default, no filtering files output
keepfilteroutput=Y ---> keepfilteroutput=Y, withfilteredset=N, filteredset=filtered.fits  // Overwrites input table (!!)
withfilteredset=N  ---> keepfilteroutput=Y, withfilteredset=N, filteredset=filtered.fits  // Overwrites input table (!!)
withfilteredset=Y  ---> keepfilteroutput=Y, withfilteredset=Y, filteredset=filtered.fits  // Writes new file
filteredset=f.fits ---> keepfilteroutput=Y, withfilteredset=Y, filteredset=f.fits         // Writes new file
Basically you have to be careful. ALL of these flags, when used in isolation, will affect their partner flags.

The parameter withfilteredset is commonly used, but it doesn't appear on evselect's doc page. It's analogous to withrateset, withimageset, etc., but it is completely disabled if keepfilteroutput=N. Therefore, keepfilteroutput is the main control on whether evselect outputs filtered results; withfilteredset just controls where filtered results are written (either modifying input table or to a new file specified by filteredset). Here are some examples

evselect keepfilteroutput=Y withfilteredset=Y filteredset=filtered.fits ...
This copies input table to filtered.fits, and modifies filtered.fits in place. The input table is not edited.
evselect keepfilteroutput=Y withfilteredset=N filteredset=filtered.fits ...
Modify the input table directly (you probably don't want to do this).
evselect table=eventlist.fits keepfilteroutput=N withfilteredset=Y filteredset=filtered.fits
This warns: "You have requested that no output be produced" and exits after doing nothing.

Thus keepfilteroutput supersedes withfilteredset. Using:

evselect keepfilteroutput=N withfilteredset=Y withhistogramset=N withimageset=N withrateset=N withspectrumset=N
will fail with a warning. The source code for evselect tests for keepfilteroutput rather than withfilteredset, when checking whether to produce output or not.

And, you can see from the example cases for filtering output above that withfilteredset is actually a redundant parameter. By using just keepfilteroutput and filteredset, you can hit all the cases you need. But I guess older code uses withfilteredset, and maybe XMM SAS folks want to maintain backwards compatibility.

Flag filtering and filtered output are pretty much independent; of course, if you dont ask for filtered output, non-destructive vs. destructive flag filtering has no effect. Note that this table (link) is confusing and I have inquired about it.

XMM-Newton event attributes

Event lists obtained from XMM-Newton MOS and PN data include a 32 bit integer FLAG column that indicates "event attributes", various characteristics (CLOSE_TO_DEADPIX, COSMIC_RAY, OUT_OF_FOV) that may render events unusable for scientific analysis. Events with undesired flags may be filtered with evselect; an example filtering expression for MOS is:

evselect expression='(PI in [200:900]) && (((FLAG & 0x766a0f63) == 0)||((FLAG & 0x766a0763) == 0))' filtertype=expression ...
This particular filter is used in the ESAS tool mos-filter to create a diagnostic soft X-ray image for MOS1 observations after XMM revolution 2383 (when CCD3 on MOS1 was disabled by a micrometeorite).

In the above example, an event with any single flag in either 0x766a0f63 or 0x766a0763 is discarded. The filters differ by only one bit, 0x800; OR'ing the two selectors yields:

0x800 & 0x766a0f63 == 0x1 != 0
0x800 & 0x766a0763 == 0x0 == 0
so this filter passes 0x800 (IN_SPOILED_FRAME) events through. And, this example is equivalent to filtering with 0x766a0763 alone.

Compared to the "standard" MOS event filter, #XMMEA_EM = 0x766ba000, our example:

This filter is used to create a soft X-ray image to help users diagnose MOS CCDs in an "anomalous" state, which appears as strong background noise below ~1 keV (Sec. 3.3, Kuntz & Snowden 2008). E.g., we want to keep OUT_OF_FOV events as enhanced corner counts are a strong indicator of an anomalous state. The reasoning for some of the other filter flags are not obvious to me, though.

Canned event filtering

The standard recommendations for MOS and PN are #XMMEA_EM and #XMMEA_EP. Ignacio de la Calle explains that the more restrictive #XMMEA_SM is no longer recommended for MOS, but filtering with FLAG==0 is recommended for PN spectral analysis (I have not explored why this is better than #XMMEA_EP). It also seems conventional to filter for single and double event patterns alongside the standard filters: PATTERN<=12 for MOS and PATTERN<=4 for PN.

This information is sourced from a Nov. 2015 helpdesk thread. One useful tidbit from this thread, which I have not seen elsewhere: 

> (1) Could you please tell me why #XMMEA_SM is not recommended any more? I
> didn't find any explanation for this online, so I want to make sure it
> clearly.

There might not be any explanation online. These flags were defined very early
in the mission by the experts in charge of the instruments. Their use and
recommendation to users comes from their experience. As of today, the use of
this flag is not recommended. But you can off course use it.

Explanations of event attributes

First, some references:

The event attribute list for MOS, taken from a processed ODF event file's FITS header: 

XMMEA_0 = '(FLAG & 0x1) != 0'  / DIAGONAL
XMMEA_1 = '(FLAG & 0x2) != 0'  / CLOSE_TO_CCD_BORDER
XMMEA_5 = '(FLAG & 0x20) != 0' / CLOSE_TO_ONBOARD_BADPIX
XMMEA_6 = '(FLAG & 0x40) != 0' / CLOSE_TO_BRIGHTPIX
XMMEA_8 = '(FLAG & 0x100) != 0' / CLOSE_TO_DEADPIX
XMMEA_9 = '(FLAG & 0x200) != 0' / CLOSE_TO_BADCOL
XMMEA_10= '(FLAG & 0x400) != 0' / CLOSE_TO_BADROW
XMMEA_11= '(FLAG & 0x800) != 0' / IN_SPOILED_FRAME
XMMEA_13= '(FLAG & 0x2000) != 0' / ON_BADOFFSET
XMMEA_15= '(FLAG & 0x8000) != 0' / FLICKERING
XMMEA_16= '(FLAG & 0x10000) != 0' / OUT_OF_FOV
XMMEA_17= '(FLAG & 0x20000) != 0' / IN_BAD_FRAME
XMMEA_19= '(FLAG & 0x80000) != 0' / COSMIC_RAY
XMMEA_21= '(FLAG & 0x200000) != 0' / ON_BADPIX
XMMEA_22= '(FLAG & 0x400000) != 0' / REJECTED_BY_GATTI
XMMEA_25= '(FLAG & 0x2000000) != 0' / OUT_OF_CCD_WINDOW
XMMEA_26= '(FLAG & 0x4000000) != 0' / OUTSIDE_THRESHOLDS
XMMEA_28= '(FLAG & 0x10000000) != 0' / ON_BADROW
XMMEA_29= '(FLAG & 0x20000000) != 0' / BAD_E3E4
XMMEA_30= '(FLAG & 0x40000000) != 0' / UNDERSHOOT
XMMEA_EM= '(FLAG & 0x766ba000) == 0' / Select good MOS events
XMMEA_SM= '(FLAG & 0xfffffeff) == 0' / Select good MOS events for spectra

Inspecting a filtered (XMMEA_EM) FITS event file shows that most remaining events have no flag (0x0). A few might have flags like 0x00000040 (decimal: 64), for attributes not thrown out by XMMEA_EM.

One example pitfall: running espfilt on an image already filtered with #XMMEA_EM will result in missing corner lightcurves, because all the OUT_OF_FOV pixels were removed (you can deduce how I learned this...).

And, the attribute list for PN: 

XMMEA_0 = '(FLAG & 0x1) != 0'  / INVALID_PATTERN
XMMEA_2 = '(FLAG & 0x4) != 0'  / CLOSE_TO_CCD_WINDOW
XMMEA_3 = '(FLAG & 0x8) != 0'  / ON_OFFSET_COLUMN
XMMEA_4 = '(FLAG & 0x10) != 0' / NEXT_TO_OFFSET_COLUMN
XMMEA_5 = '(FLAG & 0x20) != 0' / CLOSE_TO_ONBOARD_BADPIX
XMMEA_6 = '(FLAG & 0x40) != 0' / CLOSE_TO_BRIGHTPIX
XMMEA_8 = '(FLAG & 0x100) != 0' / CLOSE_TO_DEADPIX
XMMEA_11= '(FLAG & 0x800) != 0' / IN_SPOILED_FRAME
XMMEA_16= '(FLAG & 0x10000) != 0' / OUT_OF_FOV
XMMEA_17= '(FLAG & 0x20000) != 0' / IN_BAD_FRAME
XMMEA_19= '(FLAG & 0x80000) != 0' / COSMIC_RAY
XMMEA_20= '(FLAG & 0x100000) != 0' / MIP_ASSOCIATED
XMMEA_21= '(FLAG & 0x200000) != 0' / ON_BADPIX
XMMEA_22= '(FLAG & 0x400000) != 0' / SECONDARY
XMMEA_23= '(FLAG & 0x800000) != 0' / TRAILING
XMMEA_25= '(FLAG & 0x2000000) != 0' / OUT_OF_CCD_WINDOW
XMMEA_EP= '(FLAG & 0xcfa0000) == 0' / Select good PN events

I don't present the RGS tables as I'm not using RGS data currently, sorry! But they are easy to look up in your FITS header if you have the relevant files on hand.

Note that event attributes specific to an instrument (MOS, PN, RGS) are guaranteed to map to unique FLAG bits (this may have been quoted in some XMM SAS doc, but I forgot where).

E.g., the REJECTED_BY_GATTI attribute (0x00400000) references a sliding scale algorithm due to Emilio Gatti (Gatti et al. 1963, 1968, 1969) that minimizes differential non-linearity in the MOS ADC (source: xmm dfhb). The EPIC MOS Data Handling (EMDH) unit runs the GATTI correction and then applies lower and upper thresholds to events (I surmise?) before passing to the On-Board Data-Handling (OBDH) subsystem for telemetering. Events that were corrected by the Gatti algorithm, but should have been rejected by the EMDH upper threshold, are flagged as REJECTED_BY_GATTI (source: emevents doc). Point being, this is relevant only to MOS, so we are assured that PN and RGS will not use this FLAG bit for any attribute.

With this you can filter events, or decipher opaque evselect expressions, to your heart's content.