VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 1 of 45 VIRTIS-M Calibration Document Data calibration process for the Mapping channel of the VIRTIS instrument on Venus-Express NAME FUNCTION DATE PREPARED A. Cardesin VIRTIS-VEX Data Handling Engineer 26/05/2008 APPROVED S. Erard VIRTIS-VEX Archiving Manager - AUTHORIZED G. Piccioni VIRTIS-VEX Principal Investigator - VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 2 of 45 DOCUMENT CHANGE RECORD DOC TOTAL AFFECTED DESCRIPTION OF DATE ISSUE PAGES PAGES MODIFICATION 0.1 14/11/2007 17 All First draft 1.0 19/11/2007 24 All First internal release Reformatted for external release and 1.1 26/05/2008 45 All added more useful information after iteration with team VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 3 of 45 CALIBRATION SOFTWARE CHANGE RECORD SW AFFECTED DESCRIPTION OF DATE ISSUE PROCEDURES MODIFICATION 1.0 2006 All G. Piccioni: First preliminary release, internal for VIRTIS team 2.0 08/2006 All G. Piccioni: modified, internal for VIRTIS team VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 4 of 45 02 September 2006, G. Piccioni: updated some keyword for new labels 18 November 2006, G. Piccioni: modified the thermal BCKGND evolution correction for lossy compression 23 March 2007, A. Cardes?n: added error handling 27 April 2007, A. Cardes?n: added Visible ITF, added writing ITF in log file, solved minor bugs 07 June 2007, A. Cardes?n: modified INFO file: added more parameters and made PDS compatible. Delete variable raw cube before despiking 18 June 2007, G. Piccioni: set the new flag values for saturated and dead pixels, -1000 to -1004. Added the call to the destriping_selective, dead_pixels and despike_fast.pro frames_clean routines. destripe_selective.pro 28 June 2007, A. Cardes?n: Added ouput variable SOFT_VERSION (included dead_pixels.pro also in info file) 2.1 17/03/2008 frames_clean.pro 02 August 2007, A. Cardes?n: Added keyword NO_PROGRESSBAR, V_calibM_VEX.pro improved algorithm to calibrate using ITF frame by frame VEX_VIRTIS_M_VIS_ITF_DUMMY.DAT 23 August 2007, A. Cardes?n: Added new despike_fast 17 September 2007, A. Cardes?n: modified ITF filenames to 03 December 2007, A. Cardes?n: check for anomalous dark frames, modified comments 23 January 2008, A. Cardes?n: reconvert dead pixels to mis_key value at the end. Added extra check for NaN values, set to error_key=-1001. Moved Info to the end of the code and added line for Wrong NaN pixels 27 January 2008, A. Cardes?n: reorganised routines into ENVI_interface and support 11 March 2008, A. Cardes?n: corrected bug for wavelenghts in nanometers coming from previous modification VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 5 of 45 OPEN ISSUES DATE SECTION DESCRIPTION STATUS SW Change 20/03/2008 Missing information about old calibration versions Open Record 20/03/2008 1.5 Missing reference for some reference documents Open The calculation of the correct FWHM has not yet been 3.4.3, 4.3, implemented in the calibration pipeline. These values are 20/03/2008 Open 4.6.1 temporarily set to the difference between consecutive wavelengths. The calculation of the correct Uncertainty has not yet been 20/03/2008 3.4.3, 4.6.1 implemented in the calibration pipeline. These values are Open temporarily set to a dummy value of -1. An error shift of -9.8nm has been observed in the spectral 20/03/2008 4.3 registration, to be corrected in future releases of the calibration Open software. Due to some internal constraints, the radiometric response of the Visible channel has not yet been fully determined and thus 20/03/2008 4.4.1 Open the ITF file is currently a uniform image with dummy values = 1.0. A problem of different responsivity has been observed for the odd spectels (along spectral dimension) with respect to the 27/03/2008 4.3, 4.4 Open even ones. This is under investigation and is to be corrected in future releases of the calibration software. 27/03/2008 4 Description of filter crossings is to be added Open Description of straylight issues, specially for Visible channel, 27/03/2008 4 Open needs to be added Estimation of noise magnitude as a function of wavelength is 27/03/2008 4 Open to be added (e.g. dark sky) Discussion of non-nominal observations is to be added, e.g. 27/03/2008 4 single-point targets, cruise observations, high temperature Open measurements, etc VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 6 of 45 TABLE OF CONTENTS DOCUMENT CHANGE RECORD ...........................................................................................................................................2 CALIBRATION SOFTWARE CHANGE RECORD...............................................................................................................3 OPEN ISSUES ..............................................................................................................................................................................5 TABLE OF CONTENTS.............................................................................................................................................................6 1. INTRODUCTION....................................................................................................................................................................7 1.1. PURPOSE OF THE DOCUMENT ...............................................................................................................................................7 1.2. APPLICABILITY ....................................................................................................................................................................7 1.3. LIST OF ACRONYMS .............................................................................................................................................................7 1.4. APPLICABLE DOCUMENTS ...................................................................................................................................................8 1.5. REFERENCE DOCUMENTS ....................................................................................................................................................8 1.6. CONTACT INFORMATION......................................................................................................................................................9 2. VIRTIS INSTRUMENT ........................................................................................................................................................10 2.1. SCIENTIFIC OBJECTIVES.....................................................................................................................................................10 2.2. VIRTIS CHANNELS ...........................................................................................................................................................10 2.3. INSTRUMENT DESIGN ........................................................................................................................................................12 3. VIRTIS-M SCIENCE DATA FORMAT .............................................................................................................................13 3.1. TECHNICAL INTRODUCTION ...............................................................................................................................................13 3.2. GROUND DATA HANDLING ................................................................................................................................................14 3.3. RAW SCIENCE DATA FORMAT.............................................................................................................................................16 3.4. CALIBRATED SCIENCE DATA FORMAT ................................................................................................................................17 4. CALIBRATION PIPELINE FOR VIRTIS-M ....................................................................................................................22 4.1. INITIALISATION..................................................................................................................................................................22 4.2. DATA CORRECTION PRE-PROCESSING ...............................................................................................................................23 4.3. SPECTRAL REGISTRATION ..................................................................................................................................................29 4.4. RADIOMETRIC CALIBRATION .............................................................................................................................................32 4.5. DATA CORRECTION POST-PROCESSING .............................................................................................................................35 4.6. FINALISATION ....................................................................................................................................................................39 APPENDIX A ? EXAMPLE OF PDS LABEL ........................................................................................................................42 APPENDIX B ? EXAMPLE OF INFORMATION SUMMARY LOG FILE ......................................................................45 VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 7 of 45 1. INTRODUCTION 1.1. PURPOSE OF THE DOCUMENT This document gives a summarized overview of the data calibration process for the Visible and Infrared Thermal Mapping channel (-M) of the VIRTIS instrument onboard the Venus-Express mission. 1.2. APPLICABILITY This document is applicable to the M (Mapping) channel of the VIRTIS experiment onboard the Venus Express mission, both for Infrared and Visible data. 1.3. LIST OF ACRONYMS DDS Data Distribution System DC Digital Counts DN Digital Number EGSE Electronic Ground Segment Equipment ESA European Space Agency FPA Focal Plane Array FPGA Focal Plane Gain Array FWHM Full Width at Half Maximum -H High Resolution Channel HK House Keeping INAF Istituto Nazionale Astrofisica (Italy) IASF Istituto Astrofisica Spaziale e Fisica Cosmica (Rome, Italy) IR Infra Red ITF Instrument Transfer Function LESIA Laboratoire d?Etudes Spatiales et d?Instrumentation en Astrophysique (Observatoire Paris Meudon) NAIF Navigation and Ancillary Information Facility, NASA -M Mapping channel ME Main Electronics MSB Most Significant Bit PEM Proximity Electronics Module PDS Planetary Data System RD Reference Document SAP Science Activity Plan SCET Spacecraft Clock Ephemeris Time SPICE Ancillary Information System developed by NAIF VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 8 of 45 TM Telemetry VIRTIS Visible InfraRed Thermal Imaging Spectrometer VIS Visible VEX Venus Express Mission 1.4. APPLICABLE DOCUMENTS AD01 VenusExpress-VIRTIS to Planetary Science Archive Interface Control Document, VVX-LES-IC-2269, Issue 1.0, June 2007. 1.5. REFERENCE DOCUMENTS RD01 VIRTIS SW user manual, VVX-VIR-UM-001 // VVX-DLR-MA-001, Issue 3.1, July 2004 RD02 VIRTIS Instrument Description, VVX-GAF-RP-001, Issue 2.0, February 2003 RD03 VIRTIS Performance Analysis, VVX-VIR-TN-001, Issue 1.0, November 2004 RD04 VIRTIS Commissioning Report, VVX-VIR-RP-005, Issue 1.1, June 2006 (inside VEX MCRR Payloads Reports July 2006) RD05 VIRTIS Orsay On-Ground Calibration Log Book, VVX-VIRTIS_LOG_CALIB_018 / VVX-GAF-LB-008, Issue 1, March 2004 RD06 VIRTIS Orsay On-Ground Calibration Plan, VVX-GAF-PL-006, Issue 1, February 2004 RD07 On Ground Support Equipment description and performances for the calibration of VIRTIS on VENUS-EXPRESS, G. Bonello, OGSE_VVX_Orsay, draft version, June 2004 RD08 VIRTIS-M for VEX as Run Procedure (ORSAY On-Ground Calibration), VVX-GAF- TR-030?36 (multiple documents), January-February 2004 RD09 The ground calibration setup of OMEGA and VIRTIS experiments: description and performances, G. Bonello et al, Planetary and Space Sciences 53, 711-728, February 2005 RD10 VIRTIS-M Spectral Calibration, VVX-VIR-TN-002, draft version, July 2005 VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 9 of 45 RD11 VIRTIS-M Spectral Calibration and Excel files, E. Ammannito, no reference, 2005 RD12 On-ground characterization of Rosetta/VIRTIS-M. I. Spectral and Geometrical Calibrations, E.Ammannito et al., Review of Scientific Instruments 77, 093109, 2006 RD13 On-ground characterization of Rosetta/VIRTIS-M. II. Spatial and Radiometrical Calibrations, G. Filacchione et al., Review of Scientific Instruments 77, 103106, 2006 RD14 Efficiency Measurements of the VIRTIS-M grating, G. Piccioni et al., Planetary and Space Science 48, 411-417, 2000 RD15 Calibration IR Detector S/N 012 (sca84747) VIRTIS for Venus Log Report, VVX- GAF-RP-002, Issue 1, January 2004 RD16 VIRTIS Rosetta M Calibration Report. VIR-INAF-MA-002, Issue 2.0, October 2006 RD17 VIRTIS Rosetta H Calibration. VIR-DES-XX-YYYY, issue 1, October 2006 RD18 VIRTIS PDS/IDL software library VVX-LES-SW-2264, Issue 2.7.3, March 2008 RD19 VIRTIS VEX SW Internal Interface Control Document, VVX-GAF-IC-002 (attachment 6) December 2003 RD20 FM VIRTIS Experiment User Manual (Delta Documentation) VVX-VIR-UM-001, Issue 2, December 2004 RD21 VIRTIS-M FOR VEX AS RUN PROCEDURE: SPECTRAL CHARACTERISATION, VVX-GAF-TR-032, issue 1, January 2004 RD22 VIRTIS imaging spectrometer for the ESA/Venus Express mission, [5543-20] , SPIE, INFRARED SPACEBORNE REMOTE SENSING XII, Denver, August 2004 1.6. CONTACT INFORMATION Giuseppe Piccioni, VIRTIS Principal Investigator, IASF-INAF Roma (Italy) email: giuseppe.piccioni@iasf-roma.inaf.it tel: +0039-0649934445 Alejandro Cardes?n Moinelo, VIRTIS Data Handling Engineer, IASF-INAF Roma (Italy) email: Alejandro.Cardesin@iasf-roma.inaf.it tel: +0039-0649934210 VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 10 of 45 Stephane Erard, VIRTIS Data Archive Manager, LESIA Paris (France) email: stephane.erard@obspm.fr tel: +0033 (0)145077819 2. VIRTIS INSTRUMENT VIRTIS (Visible Infra Red Thermal Imaging Spectrometer) is an imaging spectrometer which allows Venus Express to map details of the Venus planet from the surface up to the thermosphere. Therefore its scientific objectives cover a large field, not only in meteorology of the middle atmosphere, as expected from optical remote sensing instruments, but also in geology, through the near infrared deep windows sounding down to the surface, and in dynamics and composition of the upper atmosphere, through non- LTE emissions of CO2 and airglow measurements. VIRTIS is constituted of three data channels in one compact instrument. Two of these channels, one visible (0.25-1?m) and one infrared (1-5 ?m) are committed to spectral mapping, and are housed in the Mapper (-M) optical subsystem. One channel is devoted solely to spectroscopy at high resolution, and is housed in the High resolution (-H) optical subsystem (2-5 ?m). Both channels usually operate simultaneously, but can also work separately, depending on observing modes. They are aligned to the boresight and have combined operations in order to provide a spectral image of 64 milliradians from the two VIRTIS-M channels, associated with one (or several) spectrum from VIRTIS-H channel. 2.1. SCIENTIFIC OBJECTIVES As a generalist instrument, VIRTIS has many scientific objectives, which are summarized below: - Study of the lower atmosphere composition (CO, OCS, SO2, H2O) from 1-2.5 ?m night side atmospheric windows - Study of the cloud structure, composition and scattering properties - Cloud tracking in the UV and IR, for retrieval of the vertical field of wind velocities - Measurements of the temperature field - Lightning search - Mesospheric sounding - Search for variations related to surface/atmosphere interaction - Temperature mapping of the surface - Search for seismic wave activity (tentative) In addition, VIRTIS also provides high definition images of Venus of great value for public outreach program. 2.2. VIRTIS CHANNELS VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 11 of 45 VIRTIS-M: Main purpose: spectral mapping visible and infrared at moderate spectral resolution. Below a list of some of the multiple science objectives of this channel both in the night and day side of the planet. - Night side o Surface mapping (using measurements at 1.02, 1.1, and 1.18 ?m) o O2 emission (at 1.27 ?m) o Lower atmospheric sounding (H2O) o Lightning search o Clouds morphology and structure o Dynamics - Day side o UV cloud absorber signature / correlation with IR absorptions o O2 emission (at 1.27 ?m), also observed at day o CO2 non LTE emissions (nadir and limb observations) o Clouds morphology o Dynamics VIRTIS-H: Main purpose: high spectral resolution for minor constituent detections and rotation/vibration structure resolved in spectral bands. Below a list of some of the multiple science objectives of this channel in both the night and day side of the planet. - Night side o 2.3 micron lower atmosphere sounding: OCS, CO, H2O measurement o thermal profile from CO2 4.3 ?m profile inversion - Day side o fluorescent emission at 4.3 micron ( nadir and limb) o Cloud IR absorptions Due to the high sensitivity of its infrared detectors, the observations with VIRTIS give a high signal to noise on the day side in a very short time; night side observations on the contrary need longer integration (up to 18s depending on the science objective), still fully compatible with orbital operations. The main difficulty for reconstructing images of the planet is the dwell time, too short at pericenter to allow VIRTIS to build images. This remark has lead the VIRTIS team to require observations at different positions along the orbit, contrary to the Mars Express observation strategy. VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 12 of 45 2.3. INSTRUMENT DESIGN The VIRTIS instrument combines a double capability: high-resolution visible and infrared imaging in the 0.25-5 ?m range at moderate spectral resolution (VIRTIS-M channel) and high-resolution spectroscopy in the 2-5 ?m range (VIRTIS-H channel). The two channels observe the same target in combined modes to take full advantage of their complementarities. VIRTIS-M (named -M in the following) is characterized by a single optical head consisting of a Shafer telescope combined with an Offner imaging spectrometer and by two two-dimensional FPAs: the VIS (0.25-1 ?m) and IR (1-5 ?m). VIRTIS-H (-H) is a high- resolution infrared cross-dispersed spectrometer using a prism and a grating. The 2-5 ?m spectrum is dispersed in 8 orders on a focal-plane detector array. Figure 1. VIRTIS instrument model VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 13 of 45 Table 1. Characteristics of the VIRTIS instrument onboard Venus Express More information on the instrument design can be found in the reference documentation (see section 1.5). 3. VIRTIS-M SCIENCE DATA FORMAT 3.1. TECHNICAL INTRODUCTION Science data generated by the VIRTIS-M VIS and VIRTIS-M IR are stream of 16bit words, each corresponding to one pixel. Detector data are acquired on a spectral basis (spectrum by spectrum). The VIS CCD detector is a frame transfer detector of 1024x1024 CCD elements, thus only 512x1024 pixels are usable as image area. Moreover, each detector element has a physical size which is the half of the IR VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 14 of 45 detector pixel, thus summation of 2x2 pixels is performed by the PEM to match spatial resolution of the IR channel. This action is performed directly by the PEM and is transparent to the final user. The ME receives from the CCD 438 spectral and 256 spatial pixels and from the M-IR in full window mode receives 438 spectral and 270 spatial pixels. For compatibility with an integer binning value these formats are further reduced by the ME to a common 432x256, which is the final format of the spectral frames that can be seen in the data. 3.2. GROUND DATA HANDLING The following table describes the VIRTIS data processing levels as used in this document. This document refers to level 3 data in CODMAC level notation, identified as type RDR. See Table 2 below. Table 2. Data processing levels The VIRTIS-VEX data set is generated by the VIRTIS-VEX team according to the following scheme: - Telemetry data packets for VIRTIS are retrieved by INAF-Rome from the VEX Data Distribution System (DDS), and are ingested into the Electronic Ground Segment Equipment (EGSE). - The EGSE performs session slicing and generates a first version of the raw data files from telemetry (preliminary level 2 data). These files are PDS-formatted and include a label, the scientific data and a selection of housekeeping parameters related to each acquisition. At this step geometric keywords and ?catalogue? keywords (related to mission operations) are not yet informed in the label. Besides, timing in the labels is based on an estimate of UTC provided by the DDS, with poor accuracy. VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 15 of 45 Figure 2. EGSE Software diagram - These preliminary raw data files are mirrored in LESIA-Meudon, where all relevant information (including label values) is ingested in a data base (Otarie). The data base also collects extra operational information from the ITL files and from the Science Activity Plan (SAP). - The information about the position of the scanning mirror is retrieved from the housekeeping packets and stored separately in the form of SPICE Camera kernels. These files are produced at periodic time intervals, usually following the medium term planning cycles defined by ESA. These files are distributed to the scientific community via the PSA inside the VIRTIS data set, and in the future also through the SPICE distribution systems (ftp://naif.jpl.nasa.gov/pub/naif/VEX and ftp://ssols01.esac.esa.int/pub/data/SPICE/VEX/kernels) - When all Spice kernels relative to a session become available, the data base triggers the geometric computation. The output of this procedure is two-fold: first it writes the geometry file corresponding to each data file. Second, it updates the raw data files by inserting the geometry and catalogue keyword values, plus the correct UTC timing, in the labels. The session-wide geometry parameters are also stored in the data base, and checked for consistency with the SAP. Index files are generated through both the PVV tool and the Otarie data base in Meudon. - The new files are mirrored back to INAF-Rome. VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 16 of 45 - Calibration procedures are then run separately for M in Rome, for H in Meudon. The calibrated data files are again mirrored at both sites, and distributed on-line to the VIRTIS science team. - At the end of the proprietary period a PDS-compliant data set is delivered to the ESA Planetary Science Archive. - Derived data products may be generated by the instrument team as a by-product of data analysis. These products will be archived in the PI institutes, and some of them may be included in the PSA distribution. Figure 3. VIRTIS pipeline diagram 3.3. RAW SCIENCE DATA FORMAT For information about the raw science data format refer to the applicable documentation: AD01, VenusExpress-VIRTIS to Planetary Science Archive Interface Control Document, VVX-LES-IC-2269, Issue 1.0, June 2007. VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 17 of 45 3.4. CALIBRATED SCIENCE DATA FORMAT The calibrated data of both sensors Visible and Infrared are stored in a format similar to the raw data, but with some important differences due to the different nature of the information (physical units in floating format) and to the different information applied and stored during the calibration process. Figure 4. Schema of the calibrated science data format with the three extra planes containing applicable information The output file includes also an ASCII header in PDS format with the most important information for the user. See AD01 for details about this PDS label and other archiving conventions used. An example of PDS label can be found on APPENDIX A ? Example of PDS LABEL (page 42). 3.4.1. Spectral cube core structure The cube core (QUBE in the notation of the Planetary Data System Standard) is a 3- dimension matrix containing the radiometrically calibrated data, named spectral radiance, stored in physical units: W m ? ?m ? sr 2 The calibrated data is saved in floating precision, using 4bytes per element. Note the difference with the raw format, which used only 2 bytes as the data was stored in digital numbers. The axes of the QUBE are called BAND (spectral band defined by the wavelength), SAMPLE (spatial direction along the slit), and LINE (acquisitions in successive time steps). VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 18 of 45 The three types of 2-dimension structures in a QUBE are called: - FRAME (fixed LINE), provided by the detector at a given time - IMAGE (fixed BAND), a spatial information reconstituted through time at a given wavelength - SLICE (fixed SAMPLE), spectra acquired through time at the same position along the slit. Figure 5. QUBE bi-dimensional data structures The three types of 1-dimension structures in a QUBE are called: - ROW (fixed BAND and LINE), spatial information acquired in a single time step at a given wavelength; - SPECTRUM (fixed LINE and SAMPLE), relative to a given footprint on the target; - SCAN (fixed BAND and SAMPLE), spatial information reconstructed through time. VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 19 of 45 Figure 6. QUBE one-dimensional data structures The atomic structure of the QUBE is called either pixel or spectel, depending whether it is considered a part of an image or a spectrum. The storage order of the QUBE is always (Band, Sample, Line) (i.e., band interleaved by pixel, or BIP) to respect the order of the telemetry data flow and to minimize data handling operations. Figure 7. Example of a real 3-dimensional VIRTIS-M Infrared observation VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 20 of 45 Figure 8. Example of one 2-D frame extracted from the previous cube Figure 9. Example of plot of single spectra extracted from the previous frame 3.4.2. Backplane The information stored in this plane corresponds to the time of acquisition (in SCET format) of each individual frame (or line). The time of acquisition considered is the centre of the integration time: t exp SCETCAL = SCETHOUSEKEEPING ? 2 This backplane is produced by the calibration pipeline and stores the time using 3 words of 2 bytes each, i.e. using a total of 6 bytes. As the backplane has the same format of the core cube (4 bytes per pixel), the time is stored with the following structure: 4 bytes for the integer part (in the first sample of the plane), and 2 bytes for the fractional part. As the instrument reads a full 2-D frame (samples x bands) on each acquisition, only one value is needed for each line of the cube, and the rest of the line remains empty (with dummy values). VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 21 of 45 256 samples Integer fractional 1st frame 4 bytes 2 bytes empty empty empty empty Integer fractional 2nd frame 4 bytes 2 bytes empty empty empty empty n lines Integer fractional 3rd frame 4 bytes 2 bytes empty empty empty empty Figure 10. Backplane structure for a high resolution cube (256 samples). Note that : 1 sample = 2 words = 4 bytes 3.4.3. Extra planes Besides the main core containing the scientific data and the exact times, every calibrated cube is stored together with another three extra planes, which are written as an extra cube in the same file. All of the planes have the same dimensions of the spectral frame, usually 432 bands per 256 samples (in the case of high resolution cubes), containing one value for each pixel/spectel on the frame. o First plane: contains the central wavelength of each pixel of the frame. This information is calculated based on the ground spectral calibration measurements (see reference documentation in section 1.5). The ?bands x samples? matrix is expressed in floating precision in physical units (?m) and is generated by the calibration pipeline (see section 3). o Second plane: contains the Full Width at Half Maximum (FWHM) of each pixel of the frame. The ?bands x samples? matrix is expressed in floating precision in physical units (same units as the wavelength data) and is generated by the calibration pipeline (see section 3). ? Open issue: The calculation of the correct FWHM has not yet been implemented in the calibration pipeline. These values are temporarily set to the difference between consecutive wavelengths (see section 4.3). o Third plane: contains the Uncertainty of the radiance value of each sample of the frame. The ?bands x samples? matrix is expressed in floating precision in physical units (same units as the radiance data) and is generated by the calibration pipeline (see section 3). ? Open issue: The calculation of the correct Uncertainty has not yet been implemented in the calibration pipeline. These values are temporarily set to a dummy value of -1. VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 22 of 45 4. CALIBRATION PIPELINE FOR VIRTIS-M The calibration pipeline for the VIRTIS-M channel is the same for both Visible and Infrared data and is performed locally in Rome. The calibration procedure can be divided in the following sections that are discussed in detail further below. 1. Initialisation 2. Data pre-processing 3. Spectral registration 4. Radiometric calibration 5. Data post-processing 6. Finalisation 4.1. INITIALISATION The first phase of the calibration pipeline is simply an initialisation of the variables for the upcoming processes. Here a summarized list of the procedures performed during this first phase: 1.1. Constant definition 1.2. Parameter extraction 1.3. Raw file loading 1.4. Dark frame removal The dark frame removal is performed using the housekeeping information stored in the cube suffix which indicates the status of the shutter for each acquisition (see AD01 and RD18 for information about the Suffix structure). The acquisitions with shutter closed (darks) are identified and removed from the cube. Figure 11. Example of infrared observation before and after the dark frame subtraction An example of IDL code for dark identification and extraction is given here below. The useful data is read into an IDL structure variable using the VirtisPDS / LecturePDS software package (see RD18) and the dark frames are identified using the flag for shutter status. VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 23 of 45 result = virtispds() ind_dark = where((result.suffix(5,0,*) and '2000'X) ne 0) ind_data = where((result.suffix(5,0,*) and '2000'X) eq 0) dark_frames = result.qube(*,*,ind_dark) ;only darks data_frames = result.qube(*,*,ind_data) ;data without darks 4.2. DATA CORRECTION PRE-PROCESSING Some of the data correction procedures regard the raw data and therefore they are performed in digital counts before the radiometric conversion. These procedures at raw level are called here ?pre-processing? methods. 4.2.1. Thermal evolution correction The instrument sensors, both infrared and visible, produce a thermally generated noise (Johnson noise) and thus the measured signal is subject to temperature changes. On the other hand, due to the high sensitivity of the infrared detectors, they usually get not only the radiation coming from the target, but also part of the signal coming from the background. This background signal comes from the cold box itself and the components of the instrument, which produce a certain level of signal depending on their temperature. The addition of this background signal and the thermal noise is added to the nominal signal coming from the target. This undesired signal, hereafter named ?dark?, is very important for the infrared detectors as it might cause saturation of the long wavelengths of the Infrared spectrum (~4- 5?m). Besides this, the fluctuations of the background signal also reduce the signal to noise ratio of the thermal part of the spectrum for the infrared channel. Dark current spectra usually have higher values in the longer wavelengths, i.e. in the thermal range, which causes some problems for the nominal observations, not only for the loss of signal to noise ratio, but also because this great signal can get to the saturation of the sensor and thus a complete loss of science in the affected region. VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 24 of 45 Figure 12. Sample of dark frame acquisition (top) and dark spectrum (bottom) Figure 13. Background signal for three different temperatures (135K, 150K, 165K) To remove the effects of both the background, the onboard software performs dark acquisitions with shutters closed every certain number of nominal acquisitions. This measured dark current is subtracted onboard before the data is downloaded to ground. VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 25 of 45 Figure 14. Dark acquisition scheme (dark rate can be modified) The dark acquisition rate is defined by the observation conditions as the dark current may vary faster depending on the temperature. In the case of very fast temperature variations, for example in pericenter measurements, the dark current subtraction performed onboard is not enough and an extra processing is needed on ground to optimise the dark current subtraction. VI0497_09 Figure 15. Venus limb infrared observation with very fast dark current variation This side effect of the dark subtraction can be easily corrected applying an interpolated correction during the ground calibration process. - for LOSSLESS compression: Correct original data values with an interpolation of the previous and next dark frames. For the lines after the last dark frame, interpolate with respect to the two previous darks VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 26 of 45 On board dark Interpolated dark subtraction subtraction Figure 16. Diagram showing difference netween on board and interpolated dark subtraction - for LOSSY compression: the same correction algorithm is used, with the difference that a smooth is applied to the interpolated dark before doing the subtraction and therefore reduce artefacts. This smooth (applied with a width of 50 pixels) is used to reduce the effect of noisy dark measurements. Figure 17. Smooth algorithm used in case of LOSSY compression. (Taken from IDL Reference Guide) A summary of this process can be found in the log file (see section 4.6.4 and APPENDIX B ? Example of information summary log file). To go back to the original data and the dark frames used, one can refer to the raw cube file. 4.2.2. Bad frames correction Whenever a single event happens in the sensor electronics, particularly in the signal pre- amplifier, it may affect a full frame acquisition, modifying not only one pixel but all the samples along the slit and the spectral dimensions. These events are easily seen in the images and may cause problems for the scientific interpretation of the data, although most of the times these ?horizontal stripes? or ?bad frames? can be recovered in the data processing corrections and the calibration pipeline. VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 27 of 45 Figure 18. VIRTIS IR image of the polar vortex (VI0473_00) before and after bad frame correction The calibration pipeline includes a standalone routine called FRAME_CLEAN (?frame_clean.pro?) that performs the correction for this type of problem. In particular the algorithm implemented in this function calculates some median and averaged values for each frame and checks whether any of this values is outside defined thresholds. If any of the frames is identified as a ?bad frame?, it is automatically replaced by an averaged interpolation of the two adjacent frames. A summary of this process can be found in the log file (see section 4.6.4 and APPENDIX B ? Example of information summary log file). Only the number of affected frames is given. To find the exact frames affected, one can refer to the raw cube file. 4.2.3. Dead pixels clean The percentage of dead pixels within the frame may increase with the time due to some environmental conditions (e.g. radiation). For this reason it is needed to keep tracking of the evolution of the detector?s ?health? and correct the dead pixel values if possible. The calibration pipeline includes a standalone routine called DEAD_PIXELS (?dead_pixels.pro?) that performs the correction for this type of problems. In particular the routine checks all the bands and identifies whenever the pixel values are much lower than a smoothed version of the same image. This dead pixels are then replaced automatically be the procedure and set to the averaged value. The routine may also serve to produce a mask of the dead pixels in the frame and even another with those ?soft? pixels that have a lower responsivity with respect to the others. This is of course very useful, not only to remove artificial artefacts from the images but also to track the evolution of the dead pixels with the time and make long term studies for the future. VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 28 of 45 Whenever the recovery of the dead pixels is not possible, for example in the case of dead adjacent pixels, the routine sets these pixels to a predefined flag, -1004. The use of this flag allows the user to identify this kind of pixels, so that they can be ignored or corrected with some user-defined processing (see section 4.6.2). A summary of this process can be found in the log file (see section 4.6.4 and APPENDIX B ? Example of information summary log file). Only the number of affected pixels is given. To find the exact pixels affected, one can refer to the raw cube file or check the predefined flag used. 4.2.4. Saturated pixels identification The data range used by the infrared and visible sensors is determined by the dynamics and the internal functioning of the main electronics of the instrument. For the VIRTIS-M channel, the data pixels in 16bit produced by the analogue-digital converter are converted to 15bit values in signed integer format and stored again in packets of 16bit. Details about the data normalization and dynamics can be found in RD01, section 10.14.2. Figure 19. Data normalization performed by the Main Electronics (see RD01, section 10.14.2 for details). As we have seen, the signal measured corresponds not only to the target of interest but also to the dark current and the thermal background radiation, and therefore the integration time has to be defined so that the total signal measured, observation + dark, are within the range values. Whenever the signal measured goes over these values, the data cannot be stored and the information is not recoverable. In general, the real threshold value used to identify saturated pixels is taken from the ground tests (D43OJ43, D43PF36, see RD05) and its value is around 24400 for the Infrared sensor and 23600 for the Visible one. The routines within the calibration pipeline do the identification and removal of the saturated pixels simply adding back the dark frames subtracted on board and checking which pixels have a value greater than the given threshold. These pixels are then set to a predefined flag: -1000. The use of this flag allows the user to identify this kind of pixels, so that they can be ignored or corrected with some user-defined processing. (see section 4.6.2). VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 29 of 45 A summary of this process can be found in the log file (see section 4.6.4 and APPENDIX B ? Example of information summary log file). Only the percentage of affected pixels is given. To find the exact pixels affected, one can refer to the raw cube file or check the predefined flag used. 4.3. SPECTRAL REGISTRATION This section gives a summary of the spectral registration for both VIRTIS-M channels, Visible and Infrared. To obtain the final curves ?band vs. wavelength?, two different on-ground calibration sessions were performed, the first one in Florence (Italy) and the second one in Orsay (France). Further details about both sessions is given here below, together with the final computations and results. 4.3.1. Subsystem Calibration, Florence The spectral profile of VIRTIS-M was first determined during the subsystem calibration in Florence by using a monochromator in scanning (see RD10 and RD11). The measurements were performed at cold case at a temperature of 135K and thus different with respect to the nominal one. Note that the values given here are not valid for the final registration as they were just a first approximation of the slopes without any reference to the dependence with temperature. To get the correct parameters of the profile, an extra calibration at different temperatures was performed in Orsay (see further below). The Figure 20 below shows the experimental points measured for some illuminated bands (black points) and the computed linear fit (red line) for both the VIS and IR focal plains. Using the relations in the figure it is possible to get for each band the value of the barycentre in nm. Figure 20. Spectral Calibration of both the focal planes. The error bars of the experimental points are too small to be detected. (Florence,RD11) VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 30 of 45 In the Table 3 below one can see a list of the parameters used in the linear fits and the value of the spectral width obtained as a mean of the measured value of the illuminated bands. Comparing the Spectral Width with ? Lambda that is the spectral distance between two consecutive bands, one can see that the measured two values are different, meaning that there is a little overlapping between two consecutive bands. VIS IR Lambda (nm) 274.399 ? 0.026 1050.54 ? 0.13 ? Lambda (nm) 1.89124 ? 0.00080 9.46673 ? 0.00071 Spectral Width (nm) 2.11 ? 0.22 10.95 ? 1.21 Table 3. Spectral profile for both M detectors: visible and infrared (Florence,RD11) 4.3.2. Orsay Calibration Only the spectral profile at a single temperature was studied in Florence, while the final spectral registration (bands vs wavelengths) was studied by using the spectral registration measurements performed at Orsay under different thermal conditions (see RD05 in section 1.5) by using the gas cells. The following table shows a summary of values obtained during the calibration measurements in Orsay, comparing reference spectra with the ones measured by VIRTIS on the Infrared channel at three different temperatures: Cold-Nominal-Hot. Table 4. Summary of the main parameters of the spectral profile for VIRTIS-M-IR. Relative table for Visible channel is not available. (Orsay, RD05) 4.3.3. Final Spectral Registration The figures ?Error! No se encuentra el origen de la referencia. show the final results of the calculations, using the values obtained in Orsay to correct the first fit obtained in Florence. In VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 31 of 45 particular one can see the final curves for temperature dependence of the slope and intercept and the parameters of the fit to be used in the final wavelength generation. Slope Intercept 1.916 288.2 1.914 288.195 y = 0.00086947x + 1.77018852 288.19 1.912 Slope [nm/band] Intercept [nm] 288.185 1.91 288.18 1.908 288.175 y = -0.00265214x + 288.59715454 1.906 288.17 1.904 288.165 1.902 288.16 288.155 1.9 130 140 150 160 170 130 140 150 160 170 Grating Tem perature [K] Grating Tem perature [K] Figure 21. Spectral registration for VIRTIS-M Visible channel (Florence+Orsay, RD11). Slope Intercept 9.505 1045 y = 0.00062407x + 9.39941505 y = -0.00991240x2 + 2.28419487x + 912.51006589 1040 9.5 Slope [nm/band] Intercept [nm] 1035 9.495 1030 9.49 1025 9.485 1020 1015 9.48 130 140 150 160 170 130 140 150 160 170 Grating Tem perature [K] Grating Tem perature [K] Figure 22. Spectral registration for VIRTIS-M Infrared channel (Florence+Orsay, RD05 + RD11). 4.3.4. Wavelength Generation Routine As a final result of the spectral calibrations performed on ground, the routine WAVELENGTH_GEN (?wavelength_gen.pro?), included in the calibration package, compiles all the necessary parameters to compute the wavelengths for both the visible and infrared channels. The slope and the intercept are calculated from the fit of the relative values following the tables above. VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 32 of 45 The only input for this routine is the information of temperature of the spectrometer (in Kelvin) and produces the ouput variables lambdaVIS and lambdaIR, which result in two vectors of 432 elements corresponding to the wavelengths in [nm] for each band. pro Wavelength_gen, Tspec, lambdaVIS, lambdaIR ; the slope and the intercept are calculated from the fit of the relative ; value during the Orsay calibration ; the slope and the intercept are calculated from the document "VVX-VIR-TN-002" lambdaVIS=fltarr(432) slopeVIS = 0.00086947*Tspec + 1.77018852 interceptVIS = -0.00265214*Tspec + 288.59715454 lambdaVIS = interceptVIS + findgen(432)*slopeVIS lambdaIR=fltarr(432) slopeIR = 0.00062407*Tspec + 9.399441505 interceptIR = -0.0099124*(Tspec)^2. + 2.28419487*Tspec + 912.51006589 lambdaIR = interceptIR + findgen(432)*slopeIR end Open issue: an error shift of -9.8nm has been observed in the spectral registration, to be corrected in future releases of the calibration software. The cause of the shift is not yet identified and it is allocated in time during the cruise phase. One possible cause may be attributed to thermal-mechanical relaxation of the instrument plus spacecraft. Open issue: a problem of different responsivity has been observed for the odd spectels (along spectral dimension) with respect to the even ones. This is under investigation and is to be corrected in future releases of the calibration software. Open issue: The calculation of the Full Width at Half Maximum (FWHM) of each pixel/spectel has not yet been implemented in the calibration pipeline. These values are temporarily set to the difference between consecutive wavelengths (see sections 3.4.3 and 4.6.1). 4.4. RADIOMETRIC CALIBRATION Raw data cubes contain the signal detected by the sensors and are expressed in Digital Numbers (DN). The counts can be converted into physical units of spectral radiance Rad (W m-2 sr-1 ?m-1) by using the following formulas: VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 33 of 45 DN (b, s, l )VIS Rad (b, s, l )VIS = tVIS ? R (b, s )VIS DN (b, s, l ) IR Rad (b, s, l ) IR = t IR ? R(b, s ) IR where b, s and l correspond to band, sample and line location of the pixel/spectel in the original cube; t is the integration time (in seconds) indicated in the PDS header of the file for VIS and IR channels; R(b,s) is the 2-dimensional responsivity matrix for VIS and IR channels. This calculation can be applied to high resolution acquisitions (432 bands times 256 samples); in other modes, when spatial and/or spectral resolutions are reduced, it is necessary to interpolate both spectral tables and responsivity matrices according to binning values. Note that both dark currents and thermal background noise are not present in the data, as they are previously subtracted by on-board processing performed in the Main Electronics and also the thermal correction processing described above (see section 4.2.1). Whenever mathematical errors happen during the calibration process, either in the ITF conversion or in any other procedure (for example in the case of division by zero), the routine sets these pixels to a predefined flag, -1001. The use of this flag allows the user to identify this kind of pixels, so that they can be ignored or corrected with some user-defined processing. (see section 4.6.2) 4.4.1. Instrument Transfer Functions The only difference between Visible and Infrared processing is the application of a specific Instrument Transfer Function for each type of data. Each of these two ITF?s is stored in a binary file as a 2-dimensional matrix, containing the values of responsivity of the sensor frame for each relative spectel. The unit of the responsivity is defined as (m? ? sr ? ?m)/(W ? s). - VEX_VIRTIS_M_IR_ITF_V2.DAT: full resolution responsivity for the VIRTIS-M infrared frame (432 bands x 256 samples). The radiometric response has been determined after the instrument calibrations performed on ground and each pixel of the matrix represents the ITF for a single spectel. VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 34 of 45 Figure 23. Graphical view of the infrared ITF matrix. - VEX_VIRTIS_M_VIS_ITF_DUMMY.DAT: full resolution responsivity for the VIRTIS-M visible frame (432 bands x 256 samples). Open issue: Due to some internal constraints, the radiometric response of the Visible channel has not yet been fully determined and thus the ITF file is currently a uniform image with dummy values = 1.0. Figure 24. Graphical view of the visible ITF matrix (currently null) 4.4.2. ITF conversion The conversion from Digital Counts (DC) to physical radiance is done following the formula given above, which in the algorithm is written as Radiance[*,*,k] = rawcube[*,*,k] / (texp*ITF_MATRIX) Where : - rawcube[*,*,k] corresponds to a single 2-D raw frame within the raw cube VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 35 of 45 - texp is the integrated time used for the observations (defined by the command parameters) - ITF_MATRIX is the 2-dimensional matrix with the responsivity values for each spectel - Radiance[*,*,k] is the 2-dimensional frame in physical units calculated from the input raw frame Note that these operations are performed after the ?pre-processing? procedures, that is, with the cube already corrected for thermal evolution, bad frames, dead pixels and saturated values (see section 4.2). Note also that the input raw cube could have negative values caused by noise and thus the output calibrated cube might also contain negative values that are not consistent. These values are not filtered by the current version of the software. 4.5. DATA CORRECTION POST-PROCESSING Some of the data correction procedures need to be performed with physical units (radiance in this case) to avoid interpreting instrumental effects as scientific valid data. Therefore these corrections are applied after the radiometric conversion and are referred here as ?post-processing? procedures. 4.5.1. Spike removal Whenever a single event is detected on one of the sensors, it is seen as a spike in the data frame, which is usually well recognised within the spectra. The most usual events affect a single pixel and can be easily recovered but in the case of heavy ions and cosmic rays, they may affect several adjacent pixels and cause problems for the data correction. Spikes Cosmic ray Figure 25. Effect of spikes and cosmic rays on a data frame of the VIRTIS M-IR sensor VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 36 of 45 Spike Figure 26. Single pixel spike in a spectrum of the M-Visible channel The calibration pipeline includes a standalone routine called DESPIKE (?despike.pro?), that performs a systematic check and corrects for single pixel spikes. ALGORITHM: i. Pixel is considered a spike if it greater/lower than the median of its local area +/- a valid margin. ii. Local area is defined as the given pixel and its 8 neighbours along the frame (sample + bands dimension) iii. Valid margin is defined by the standard deviation of the local area multiplied by a certain level (usually 3.0). iv. The standard deviation of the local area is defined as : (second highest - second lowest pixels in the local area), divided by 2 v. When cube contains NaN values, spikes next to NaN pixel are not detected. Figure 27. Local area of a single pixel is defined as the 9 adjacent pixels along frame dimension. Since the Calibration Software version 2.1, a new version of the despiking routine is used, DESPIKE_FAST (?despike_fast.pro?). This new version applies the algorithm at frame scale, that is, not going pixel by pixel, but applying the local areas globally for each frame (see source code for more details). In principle the base of the algorithm is the same avoiding loop cycles and is up to five times faster than the old one with the same results. In some cases there might be small differences with respect to the old despike procedure due to NaN values and VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 37 of 45 the accuracy of IDL float numbers, specially when using large pixel values (raw IR/VIS or calibrated VIS cubes). Note that the current despike routine is used systematically for all files and it uses a method implemented specifically for VIRTIS data, which is meant to reduce the effect of spikes trying to preserve the most of the scientific information. There are many other noise reduction routines that can be used to remove spikes, but they have not been implemented in the pipeline to avoid loosing valid data as much as possible. The user is free to use any other noise reduction routines if desired, depending on the scientific use of the data. On the other hand we cannot exclude that some scientific information could be lost during the current despiking process. Although it is very unlikely, the current procedure could cause a lose of valid information for very specific cases with data being focused on a single spectel of the frame. For these cases the user should refer to the raw data and perform a specific calibration without despiking. A summary of this process can be found in the log file (see section 4.6.4 and APPENDIX B ? Example of information summary log file). Only the percentage of affected pixels is given. To find the exact pixels affected, one can refer to the raw cube file. 4.5.2. Stripe removal In the scientific observations performed by VIRTIS we can often see vertical stripes on each band, that is, samples that for a certain reason have a different responsivity with respect to the other ones along the slit, without being reflected in the ITF. The calibration pipeline includes a standalone routine called DESTRIPING_SELECTIVE (?destriping_selective.pro?), that performs the correction for this problem. ALGORITHM: The algorithm implemented in this routine has 4 separate steps that work for each spectral band in the cube file: i. Identify the data that are within a given valid range, e.g. to exclude deep space, saturated values, etc ii. For each separate band, make a mask of lines to define valid regions, avoiding deep space, saturated values, dead pixels, etc. The defined mask is always rectangular and covers a certain number of lines for each band. VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 38 of 45 MASK Figure 28. Illustration of the masking process iii. For each separate band, calculate the average of each column in the previously defined mask Average of each column Figure 29. Illustration of the averaging of each column iv. For each separate band, apply the correction factor of each column. This factor is calculated from the difference of the average of each column with respect to the adjacent ones. The order of magnitude of the correction is around 2~3% of the measured radiance. RESULTS: The results of this process for different kind of images can be seen in the following figures. As one can see the effect of the vertical stripes is reduced improving the scientific quality of the images without losing any information in the spectral dimension. VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 39 of 45 Figure 30. Effect of destriping process on image of the lower clouds in night side Figure 31. Effect of destriping process on image of the top level atmosphere in day side 4.5.3. Spectral filtering The spectral filtering is not performed in the present version of the calibration process. 4.5.4. Spatial filtering The spatial filtering is not performed in the present version of the calibration process. 4.6. FINALISATION This section of the process includes all the procedures performed at the end of the calibration process, once the calibrated cube is in its final format. This section includes the finalization of the cube, generation of the extra information and the creation of the output files. VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 40 of 45 4.6.1. Extra information As introduced in the previous section 3.4.3, the calibration process generates not only the radiometrically calibrated data, but also some extra information for the scientific interpretation of the data. This information is generated in the form of three planes with the same dimensions of the spectral frame, usually 432 bands per 256 samples (in the case of high resolution cubes), containing one value for each pixel/spectel on the frame. The three planes contain the following information: o First plane: central wavelength of each pixel/spectel of the frame (in microns). See section 4.3 Spectral registration for more information about the calculation of this values. o Second plane: Full Width at Half Maximum (FWHM) of each pixel/spectel of the frame. ? Open issue: these values are temporarily set to the difference between consecutive wavelengths. The calculation of the correct FWHM has not yet been implemented in the calibration pipeline (see section 3.4.3 and 4.3.3). o Third plane: Uncertainty of the radiance value of each sample of the frame. ? Open issue: these values are temporarily set to a dummy value of -1. The calculation of the correct uncertainty has not yet been implemented in the calibration pipeline. 4.6.2. Flag special pixels During the whole of the calibration process, some of the pixels are identified as invalid data, that is, without any meaningful or representative scientific information. These pixels or spectels, depending on the case, are set to a certain predefined flag representing either the cause of the error or the reason for invalidity. The utilisation of these flags allows the final user to identify this kind of pixels, so that they can be ignored or corrected with some user- defined processing. As a general rule, the user should know that any pixel with value below -999 can be considered invalid. The list of predefined flags and their meaning is given in the following table. Flag PDS keyword Meaning value Minimum valid value. Data below this value has no scientific CORE_VALID_MINIMUM -999 meaning and must be considered a flag. VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 41 of 45 Used in the case of saturated values. CORE_HIGH_INSTR_SATURATION -1000 (see section 4.2.4) Used in the case of mathematical errors happened during the CORE_HIGH_REPR_SATURATION -1001 calibration process, e.g. division by zero. (see section 4.4.2) CORE_LOW_INSTR_SATURATION -1002 currently not used CORE_LOW_REPR_SATURATION -1003 currently not used Null value. Used for missing data pixels, that is for dead or CORE_NULL -1004 defective pixels which could not be recovered. (see section 4.2.3) Table 5. Predefined flags used for identification of invalid values 4.6.3. Write calibrated cube file At the end of the calibration process and once all the data and extra information is available, the routine V_CONVLABEL (?v_convlabel.pro?) is called to write the output calibrated file in its final PDS format. The routine takes as inputs the radiometrically calibrated data cube and the extra information planes produced in the previous steps of the process, together with the time information read from the raw housekeeping. All this is used by the routine to create an output binary file following the format discussed in the section 3.4. This VI0046_00.QUB VI0046_00.CAL The output file includes also an ASCII header in PDS format with the most important information for the user. See AD01 for details about this PDS label and other archiving conventions used. An example of PDS label can be found on APPENDIX A ? Example of PDS LABEL (page 42). 4.6.4. Write Info log file Right after the creation of the calibrated cube file, the software generates an ASCII log file with the summary of the calibration process and the most important information of the whole pipeline. This file is saved in the same path as the calibrated cube, and uses the same base name with the extension ?.TXT?. For example: ...\VIR0046\CALIBRATED\VI0046_00.CAL Calibrated cube file ...\VIR0046\CALIBRATED\VI0046_00.TXT Summary info log file In the APPENDIX B ? Example of information summary log file (page 45) one can see an example of the contents of one info log file, with the PDS header for compliancy with the archiving standards. Note that the format of these files can be modified in future versions of the calibration software. VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 42 of 45 APPENDIX A ? EXAMPLE OF PDS LABEL PDS_VERSION_ID = PDS3 LABEL_REVISION_NOTE = "SE, 18/06/2007" /* File format and length */ PRODUCT_ID = "VI0046_00.CAL" RECORD_TYPE = FIXED_LENGTH RECORD_BYTES = 512 FILE_RECORDS = 100351 LABEL_RECORDS = 13 FILE_STATE = CLEAN /* Pointers to data objects */ ^HISTORY = 14 OBJECT = HISTORY DESCRIPTION = "Reserved area for ISIS compatibility" END_OBJECT = HISTORY ^QUBE = 15 ^QUBE = 2607 /* Producer information */ PRODUCER_ID = VEX_VIRTIS_TEAM PRODUCER_FULL_NAME = "DROSSART-PICCIONI" PRODUCER_INSTITUTION_NAME = "ISTITUTO NAZIONALE ASTROFISICA" PRODUCT_CREATION_TIME = 2008-03-12T01:30:59.00 TELEMETRY_SOURCE_ID = "VIRTIS_EGSE_3" SOFTWARE_VERSION_ID = {"VirtisVEX SW v.2.50", "EGSE_SOFT_7.0", "PDS_CONVERTER_7.0","EGSE2PSA_CONVLABEL_1.2.1", "GEOVIRTIS_1.9", "V_GEOLABEL_2", "GEOVIRTIS_1.9", "V_GEOLABEL_2", "GEOVIRTIS_1.9", "V_GEOLABEL_2", "GEOVIRTIS_3.3", "V_GEOLABEL_3", "V_CALIBM_VEX_2.1", "V_CONVLABEL_1.3"} /* Data description parameters */ DATA_SET_NAME = "VENUS EXPRESS VENUS VIRTIS 2/3 V1.0" DATA_SET_ID = "VEX-V-VIRTIS-2/3-V1.0" RELEASE_ID = 0001 REVISION_ID = 0000 PRODUCT_TYPE = RDR PROCESSING_LEVEL_ID = 3 STANDARD_DATA_PRODUCT_ID = "VIRTIS DATA" MISSION_NAME = "VENUS EXPRESS" MISSION_ID = VEX INSTRUMENT_HOST_NAME = VENUS_EXPRESS INSTRUMENT_HOST_ID = VEX MISSION_PHASE_NAME = "PHASE 1" PI_PDS_USER_ID = "DROSSART-PICCIONI" INSTRUMENT_NAME = "VISIBLE AND INFRARED THERMAL IMAGING SPECTROMETER" INSTRUMENT_ID = "VIRTIS" INSTRUMENT_TYPE = "IMAGING SPECTROMETER" ^INSTRUMENT_DESC = "VIRTIS_EAICD.TXT" VEX:CHANNEL_ID = "VIRTIS_M_IR" PHOTOMETRIC_CORRECTION_TYPE = NONE DATA_QUALITY_ID = 1 DATA_QUALITY_DESC = "0:INCOMPLETE ; 1:COMPLETE" /* Science operations information */ TARGET_TYPE = "PLANET" TARGET_NAME = "VENUS" START_TIME = 2006-06-05T16:46:47.450 STOP_TIME = 2006-06-05T16:51:44.483 SPACECRAFT_CLOCK_START_COUNT = "1/00039890807.13416" SPACECRAFT_CLOCK_STOP_COUNT = "1/00039891104.16757" ORBIT_NUMBER = 46 VEX:SCIENCE_CASE_ID = 2 VEX:SCIENCE_CASE_ID_DESC = "Please refer to VEX_SCIENCE_CASE_ID_DESC.TXT" VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 43 of 45 OBSERVATION_TYPE = "NULL" SPACECRAFT_ORIENTATION = (-0.0000,0.0000,1.0000) ^SPACECRAFT_ORIENTATION_DESC = "VEX_ORIENTATION_DESC.TXT" SPACECRAFT_POINTING_MODE = "NADIR_POW" ^SPACECRAFT_POINTING_MODE_DESC = "VEX_POINTING_MODE_DESC.TXT" DECLINATION = -999.999 RIGHT_ASCENSION = -999.999 MAXIMUM_LATITUDE = -42.9705 MINIMUM_LATITUDE = -76.6521 EASTERNMOST_LONGITUDE = 257.389 WESTERNMOST_LONGITUDE = 167.741 SLANT_DISTANCE = 63522.2 /* Instrument status */ INSTRUMENT_MODE_ID = 19 ^INSTRUMENT_MODE_DESC = "VIRTIS_EAICD.TXT" INST_CMPRS_NAME = "REVERSIBLE" INST_CMPRS_RATE = "N/A" INST_CMPRS_RATIO = 2.57300 VEX:VIR_IR_START_X_POSITION = 1 VEX:VIR_IR_START_Y_POSITION = 7 VEX:SCAN_MODE_ID = 1 SCAN_PARAMETER = (-30.2126,2.8333,0.2582,1) SCAN_PARAMETER_DESC = ("SCAN_START_ANGLE","SCAN_STOP_ANGLE", "SCAN_STEP_ANGLE","SCAN_STEP_NUMBER") SCAN_PARAMETER_UNIT = ("DEGREE","DEGREE","DEGREE","DIMENSIONLESS") FRAME_PARAMETER = (0.02,1,2.5,20) FRAME_PARAMETER_DESC = ("EXPOSURE_DURATION","FRAME_SUMMING", "EXTERNAL_REPETITION_TIME","DARK_ACQUISITION_RATE") FRAME_PARAMETER_UNIT = ("S","DIMENSIONLESS","S","DIMENSIONLESS") EXPOSURE_DURATION_DESC = "-1: many exposure times (calibration sessions) Values are available in sideplane. See EAICD.TXT" MAXIMUM_INSTRUMENT_TEMPERATURE = (86.4975,155.3580,154.6790,75.111,165.05) INSTRUMENT_TEMPERATURE_POINT = ("FOCAL_PLANE","TELESCOPE", "SPECTROMETER", "CRYOCOOLER","VIS_FOCAL_PLANE") INSTRUMENT_TEMPERATURE_UNIT = ("K","K","K","K","K") /* Pointer to navigation data files*/ SPICE_FILE_NAME = {"NAIF0008.TLS","PCK00008.TPC","DE405S.bsp", "CLOUD-VENUS.ker","venus_cloud.bsp","VEX_070121_STEP.TSC","VEX_V06.TF", "VEX_VIRTIS_V03.TI","ORHV_______________00030.bsp", "ORVV__060601000000_00054.bsp","ATNV_P051109051109_00124.bc", "VEX_VIRTIS_060603_060701_01.bc"} /* Spectral Reference Cube */ OBJECT = QUBE AXES = 3 AXIS_NAME = (BAND,SAMPLE,LINE) CORE_ITEMS = (432,256,3) CORE_ITEM_BYTES = 4 CORE_ITEM_TYPE = "REAL" CORE_BASE = 0.0 CORE_MULTIPLIER = 1.0 CORE_VALID_MINIMUM = "NULL" CORE_NULL = "NULL" CORE_LOW_REPR_SATURATION = "NULL" CORE_LOW_INSTR_SATURATION = "NULL" CORE_HIGH_REPR_SATURATION = "NULL" CORE_HIGH_INSTR_SATURATION = "NULL" CORE_NAME = ("WAVELENGTH","FWHM","UNCERTAINTY") CORE_UNIT = ("MICRON","MICRON","W/m**2/sr/micron") SUFFIX_BYTES = 2 SUFFIX_ITEMS = (0, 0, 0) END_OBJECT = QUBE /* Data Cube keywords */ OBJECT = QUBE VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 44 of 45 AXES = 3 AXIS_NAME = (BAND,SAMPLE,LINE) CORE_ITEMS = (432,256,113) CORE_ITEM_BYTES = 4 CORE_ITEM_TYPE = "REAL" CORE_BASE = 0.0 CORE_MULTIPLIER = 1.0 CORE_VALID_MINIMUM = -999 CORE_NULL = -1004 CORE_LOW_REPR_SATURATION = -1003 CORE_LOW_INSTR_SATURATION = -1002 CORE_HIGH_REPR_SATURATION = -1001 CORE_HIGH_INSTR_SATURATION = -1000 CORE_NAME = RADIANCE CORE_UNIT = "W/m**2/sr/micron" SUFFIX_BYTES = 2 SUFFIX_ITEMS = (1,0,0) BAND_SUFFIX_NAME = "SCET" BAND_SUFFIX_UNIT = DIMENSIONLESS BAND_SUFFIX_ITEM_BYTES = 2 BAND_SUFFIX_ITEM_TYPE = MSB_UNSIGNED_INTEGER BAND_SUFFIX_BASE = 0.0 BAND_SUFFIX_MULTIPLIER = 1.0 BAND_SUFFIX_VALID_MINIMUM = 0 BAND_SUFFIX_NULL = 65535 BAND_SUFFIX_LOW_REPR_SAT = 0 BAND_SUFFIX_LOW_INSTR_SAT = 0 BAND_SUFFIX_HIGH_REPR_SAT = 65535 BAND_SUFFIX_HIGH_INSTR_SAT = 65535 END_OBJECT = QUBE END VIRTIS Doc. Issue : : VVX-VIR-RP-008 1 for Rev. : 1 Date : 26/05/2008 Page : 45 of 45 APPENDIX B ? EXAMPLE OF INFORMATION SUMMARY LOG FILE PDS_VERSION_ID = PDS3 RECORD_TYPE = STREAM OBJECT = TEXT INTERCHANGE_FORMAT = ASCII PUBLICATION_DATE = 2008-03-12 NOTE = "Info summary for the calibration process" END_OBJECT = TEXT END START INFO ---Info summary for the calibration process--- Calibration executed on : 2008-03-12T01:30:59 Calibration routine used : V_calibM_VEX_2.1 Instrument Transfer Function used : VEX_VIRTIS_M_IR_ITF_v2.DAT FILENAME : V:\DATA\MTP002\VIR0046\RAW\VI0046_00.QUB Data are from (CHANNEL_ID) : VIRTIS_M_IR Data format : 432x256x113 (BxSxL) Instrument mode : 5-ALL_PIX_FULL_WIN Repetition (EXTERNAL_REPETITION_TIME) : 2.50 s Visible exposure time : 0.360000 s (formal) 0.360050 s (used for cal.) Infrared exposure time : 0.020000 s (formal) 0.020050 s (used for cal.) Visible delay time : 0.100000 s Infrared delay time : 0.260000 s Scan mechanism mode (SCAN_MODE_ID) : SCAN Scan step angle (SCAN_STEP_ANGLE) : Not available Summing (FRAME_SUMMING) : SUMMING OF 1 Compression mode (ENCODING_TYPE) : "REVERSIBLE" at rate "N/A" START_TIME : 2006-06-05T16:46:47.450 STOP_TIME : 2006-06-05T16:51:44.483 Average spectrometer temperature : 152.946 K (used by the wavelength gen routine) Sigma spectrometer temperature : 0.0166291 K Intercept used for the wavelength list: 1.029993 micron Slope used for the wavelength list : 0.009495 micron Dark rate : 20 Number of dark frames : 6 ----------Notes---------- Exposure time consistent in header Visible and Infrared exposures are centered within : 1.00E-002 s (+ means VIS after IR) The thermal background correction has been performed The frames clean procedure has been performed and 0 frames out of 119 were cleaned up Infrared saturation level : 24400 (including dark, evaluated from D43OJ43) The total number of pixels saturated is 0.00776569% (replaced with value: -1000) The fast despiking has been performed using threshold 3.0 and 0.83537544% of pixels have been removed The destriping has been performed on 405 bands; error computed on 83.8147% of the pixels A total of 179 defective pixels have been found in the frame: 37 dead pixels were cleaned and corrected. 142 pixels could not be corrected and were set to value - 1004.00 The spectral filtering has NOT been performed The spatial filtering has NOT been performed A total of 0 wrong pixels became NaN/Inf during the calibration process and were set to value -1001.00 ------------------------- END INFO