Combined measurements Report for the period 18. May 2010 Table of Content 1. Timeline 2. COPS, DFMS & RTOF combined measurements 1. Timeline 18. 5., out of pass DFMS, COPS & RTOF combined measurements 2. COPS, DFMS & RTOF combined measurements This was a big milestone for Rosetta with all three sensors measuring scientifically together for the first time. Except for one sensor error (DFMS MCP back voltage), an error which happens from time to time and which is fully understood, everything worked perfectly. All science data were received and showed an excellent performance of all three sensors. It is clear that the RTOF parameters have to be refined in the lab during the hibernation phase to get the maximum performance out of this sensor. This will need a SW patch in 2014. ROSINA Lutetia flyby Report for the period 7. July.-11. July 2010 Table of Content 1. Timeline 2. Performance of COPS & DFMS 3. RTOF 4. DPU 5. Anomaly reports 6. Conclusions 7. Annex 1: 1. Timeline Date Event Remarks 7.7. DFMS &COPS background measurements and heating of DFMS ion source 8.7. DFMS &COPS background measurements and heating of DFMS ion source 1 Filament bias error, probably due to a HK readout error 9.7. DFMS &COPS background measurements and heating of DFMS ion source 10.7. 0600h Switch on of DFMS and COPS 0800h Switch on of RTOF 11:48 RTOF switch-off due to a readout problem between DPU and RTOF RTOF is switched on automatically 12:06 RTOF switch-off due to a readout problem between DPU and RTOF RTOF stays off 13:54 DPU reboot with inverted instrument status COPS, DFMS remain off, RTOF is switched on 18:13 Switch-off of RTOF (commanded) Error reports because the switch-off was commanded while RTOF was still measuring 11. 7. Switch-off of ROSINA 2. Performance of COPS & DFMS Up to the reboot of the DPU on 10.7., 13:54h COPS and DFMS worked very nominally for more than three days. One error event concerning the filament bias which is most probably due to a HK hickup and from which DFMS recovered by itself happened on July 8. Unfortunately the DPU reboot switched both sensors of on July 10, 13:54h. COPS was switched on to stby shortly before the final switch-off of ROSINA on July 11, 01:45h. 3. RTOF RTOF was switched on on July 10, 08:00h. It was clear that there is some risk involved due to the fact that we had very little testing time for the operation of all three sensors due to the RTOF HV anomalies. RTOF worked flawlessly until 12:40h when the data readout got stuck. It was switched off and on again by the DPU. At 12:06 the same error happened again. It was switched off and because it was the same error as before, not switched on again. The cause of these errors is unknown. Due to the fact that they happened during the flip of the SC where a lot of subsystems were active it could be due to some interference. The reboot of the DPU at 13:54h switched RTOF on again which then worked flawlessly until it was switched off. However, due to the switch-on which needs appr. 20 minutes the switch-off came too early before RTOF finished the mode which led to error reports during the switch-off. 4. DPU The DPU reboot at 13:54 is most probably due to a watchdog alert. However, the cause is unknown (SEU?). The reason for the loss of COPS science data is being investigated. A detailed report and a timeline of events is attached (Annex 1). 5. Anomaly reports One anomaly report was raised: ID: ROS_SC-202 Rosina: unexpected behavior of sensors during Lutetia 6. Conclusions It was clear from the beginning that there was a certain risk involved in running all three ROSINA sensors together as this has been done only once during PC12, never before. However, it was felt that, while the chances of detecting the Lutetia exosphere were slim, we could gain a lot of experience with running all three sensors. This has bee proven to be correct. Therefore, when RTOF switched off this was somewhat expected. What was certainly not expected was the switch-off of DFMS and COPS. It is clear that the DPU SW has to become more failure tolerant without enhancing the risks to the sensors as main controller resets, DPU reboot, etc may happen again. On the other side, the high voltage of RTOF seems no longer to be a problem. Because DFMS and COPS were switched off by the DPU in a rather unfriendly way we would very much appreciate if we could switch on these sensors again before going into hibernation to verify their integrity. There was a SW patch foreseen to correct the RTOF cover problem, followed by a cover exercise. We therefore suggest that we upload a patch which not only solves the problem of the cover, but also the problem of the inverted instrument status after reboot and the problem of the failure handling if a sensor fails due to a reset. After that the cover operations have to be done for RTOF (during interactive pass) followed by a non-interactive operation of DFMS and COPS (<2h). From a scientific point of view we are very happy that we have valuable science data throughout the flyby. RTOF has proven to deliver excellent mass spectra although it is not yet fully optimized. These data are complemented by DFMS and COPS data which helps to connect the Lutetia flyby and the Lutetia rehearsal data. Data analysis is ongoing, but we may have a tentative detection of the Lutetia exosphere. It is clear that the data gained during the Lutetia rehearsal are extremely valuable in distinguishing between SC outgassing and real exosphere data. Before coming to a firm conclusion more work on data reduction, data uncertainties, etc.has to be done. 7. Annex 1: Order of main events: The second automatic switch-off of RTOF and no automatic switch-on after RTOF MC-Reset error was expected since the same error occurred again. At about 191.13:54 a second order error, the complete reboot of DPU occurred for the first time during hole mission. This was probably caused by an automatic watch-dog reset. After reboot the DPU tried to restore the same configuration as it was before. Unfortunately, because of the RTOF errors and switch-off before, the status of sensors was recovered inverted. This resulted in switch-on of RTOF but no switch-on of COPS and DFMS (although th e HK reports seems to indicate something different, all sensors were switched-off during reboot of DPU). This behavior could be reproduced on ground. The context file and primary current consumption was verified in detail and everything is as expected and consistent with the last HK packet received before switch-off. All other errors are only minor or subsequent errors and will be taken into account during revision of error handling. Main action: The overall concept of autonomous error handling will be revised, simplified and made fault-tolerant to cope with complex operations of all three sensors in parallel. Currently if a mode/seq. command is received for one sensor and it is off, an error is generated (good for interactive operation). This will be revised such that a mode/seq. command will switch on the sensor needed automatically. The potential problem of continuous switch off/on for persistence errors will be avoided by introducing a time window after automatic switch on. Additionally, automatic restoration of configuration after DPU reboot will be changed to handle all possible configurations, independent of what happened before. Minor improvements: - DFMS IS heater mode repeat - Switch on subsystems automatically after error (e.g. filament, TBC) - Abort RTOF ETS acquisition and readout automatically after short waiting time - Wait until ETS completely read out before switching off ETS - Decrease number of DARO errors (65 -> 5) - Check DPU mode settings for all possible configurations - Check COPS science data processing and sending to SC Date|Time (UTC)|TM Report|Unit|Event|Parameters|Remarks 7.7.|0:00:20|POST report| | | |Rosina Switch On, everything nominal | |Mode Change Event|DPU | |Change to Mode D4|Unsynchronised time packets received only at 06:14, remaining from last switch-off, OK| |3 Progress Events|All 3|PRO_SENOFF| | |4:00:37|HK report| | |Time synchronised|OK |4:01:20|3 Progress Events|All 3|PRO_SENOFF| |OK |4:05:25|Mode Change Event|COPS| |Change to Mode A5|COPS switched on | |Sensor On Event|COPS| | | |4:14:23|Mode Change Event|DFMS| |Change to Mode A2|DFMS switched on | |Sensor On Event|DFMS| | | | |Progress Event|DFMS|DM_RDPH_ON| |DFMS Heater On |4:28:50|Progress Event|COPS|PRO_C_LEVEL| |COPS NG operational level reached |4:29:32|Progress Event|DFMS|PRO_D_HVINIT| |DFMS HV Initialized |4:29:33|Progress Event|DFMS|PRO_D_FILA| |DFMS Filament Initialized |4:31:41|Science Data|DFMS| | |First DFMS science data received |Everything runs very nominally| | |Standard progress reports are omitted | |4:47:41|Progress Event|COPS|PRO_C_LEVEL| |COPS RG operational level reached |4:57:04|Progress Event|COPS|PRO_C_LEVEL| |COPS NG and RG finally adjusted | | | | | | 8.7.|9:06:22|HK report 1760| | |SID: 11, Seq Cnt: 3502| |9:14:22|HK report 1768| | |SID: 11, Seq Cnt: 3518|7 missing HK packets, no DPU malfunction detectable |17:22:24|Sensor Error Event|DFMS|HK Mon|DM_FI1BIAS, Setvalue: -45V, Read: 0x1b (=0V)|Filament switched off automatically |18:09:33|Progress Event|DFMS|PRO_D_FILA| |Filament switched on again after mode cmd received |22:29:20|Progress Event|DFMS|PRO_SENOFF| |DFMS switched off | | | | | | 9.7.|3:14:25|Mode Change Event|DFMS| |Change to Mode A2|DFMS switched on | | | | | | 10.7.|1:28:21|Progress Event|DFMS|PRO_SENOFF| |DFMS switched off |6:14:26|Mode Change Event|DFMS| |Change to Mode A2|DFMS switched on |7:59:20|Mode Change Event|RTOF| |Change to Mode A4|RTOF switched on |7:59:26|Mode Change Event|RTOF| |Change to Mode A1|Wrong mode setting, needs to be changed |8:04:24|Progress Event|RTOF| | |RTOF HV switched on | | | | | |Parallel operation of DFMS and RTOF |11:38:04|Progress Event|RTOF|PRO_R_ETSACQ| |Last event indicating RTOF ETS readout |11:40:53|HK report| | |Memory DM: 58% remains high (normal 14%)|RTOF ETS readout and science data processing gets stuck, needs fault tolerant procedure to reset automatically |11:48:25|Progress Event|RTOF|RM_STATE8| |First event indicating that RTOF don't respond to HK request |11:48:35|Sensor Error Event|RTOF|HK Mon|RM_P5V, Setvalue: 5V, Read: 0xFFFF |Error since RTOF still don't respond | |Progress Event|RTOF|PRO_SENOFF| |RTOF switched off automatically |11:48:38|2 DPU Error Event|RTOF|ERR_CMDFIFO| |Error in command FIFO processing of RTOF since unit was switched off, expected subsequent error | |HK report| | |Memory DM: 14%|RTOF science memory discarded |11:48:52|Sensor On Event|RTOF| | |RTOF switched on again automatically, starting with same mode (511) as before |11:49:11|Progress Event|RTOF| | |RTOF HV switched on |11:49:48|HK report| | |ETS Acqu Stat Overflow, Read Stat. On|Indicating ETS readout problems before (after 11:38) |12:06:23|Progress Event|RTOF|PRO_R_ETSACQ| |RTOF ETS readout finished after resuming |12:08:26|Progress Event|RTOF|RM_STATE8| |First event indicating that RTOF don't respond to HK request |12:08:30|Sensor Error Event|RTOF|HK Mon|RM_P5V, Setvalue: 5V, Read: 0xFFFF |First error event since RTOF still don't respond | |Progress Event|RTOF|PRO_SENOFF| |RTOF switched off automatically |12:09:20|DPU Error Event|RTOF|ERR_CMDFIFO| |Error in command FIFO processing of RTOF since unit was switched off, expected subsequent error | | | | | |RTOF NOT switched on again automatically since same error happened again |13:05:17|DPU Error Event|RTOF|ERR_CMDFIFO| |Error in command FIFO processing of RTOF since unit was switched off, expected subsequent error |13:24:20|DPU Error Event|RTOF|ERR_CMDFIFO| |Error in command FIFO processing of RTOF since unit was switched off, expected subsequent error |13:33:29|COPS NG and RG Science data packet missing| | | |Further investigation ongoing |13:43:29|COPS NG and RG Science data packet missing| | | | |13:53:33|Last TM Packet from Rosina| | | | |0:00:00|Power On Self Test Event| | | |Time sync lost (expected) |0:00:52|First HK packet| | |All counters zero (expected), DPU Mode A110 |Complete reboot of DPU, probably caused by an automatic watch-dog reset. Because of the RTOF switch-off before, the status of sensors was recovered inverted: DFMS, COPS off, RTOF on |0:05:54|Last HK packet without time| | | | |14:01:26|First HK with time sync again| | | |Time diff: 474s - 414s without sync - 30s boot = 30s (20s watch-dog plus delta) -> OK |14:07:43|Sensor Error Event|COPS|HK COPSIMM|TableID:4, LimitID: 0|Since COPS is off, no functionality |14:18:20|Acceptance Report|RTOF| | |First RTOF command received after switch on |14:19:25|Progress Event|RTOF| | |RTOF HV switched on |18:09:36|Progress Event|RTOF|PRO_R_ETSACQ| |RTOF ETS readout finished |18:09:43|Sensor Error Event|RTOF|ETS DARO|TableID:1, LimitID: 5| |18:09:45|HK report| | |RTOF Mode 1, ETS power already off, ETS Read Status still ON| |18:09:47|Sensor Error Event|RTOF|ETS DARO|TableID:2, LimitID: 5| |18:13:30|Sensor Error Event|RTOF|ETS DARO|TableID:65, LimitID: 5|65 Error events because RTOF ETS was switched off too early when readout was still not completed (mode 1 command on stack) |18:18:29|Progress Event|RTOF| | |RTOF in standby | | | | | | 11.7.|1:44:19|Progress Event|RTOF|PRO_SENOFF| |RTOF switched off |1:45:19|Mode Change Event|COPS| |Change to Mode 05|COPS switched on because of standard SC procedure | |Sensor On Event|COPS| | | |1:50:20|Progress Event|COPS|PRO_SENOFF| |COPS switched off |1:50:22|Mode Change Event| | |Change to Mode D2|Rosina ready to switch off