14 Jun 2013 - CronJob ran 1490/1491 ----------- 13 Jun 2013 - CronJob ran 1489/1490 ----------- 12 Jun 2013 - CronJob ran 1488/1489 ----------- 11 Jun 2013 - CronJob ran 1487/1488 ----------- 10 Jun 2013 - CronJob ran 1486/1487 ----------- 09 Jun 2013 - CronJob ran 1485/1486 ----------- 08 Jun 2013 - CronJob ran 1484/1485 ----------- 07 Jun 2013 - CronJob ran 1483/1484 ----------- 06 Jun 2013 - CronJob ran 1482/1483 ----------- 05 Jun 2013 - CronJob ran 1481/1482 ----------- 04 Jun 2013 - CronJob ran 1480/1481 ----------- 03 Jun 2013 - CronJob ran 1479/1480 ----------- 02 Jun 2013 - CronJob ran 1478/1479 ----------- 01 Jun 2013 - CronJob ran 1477/1478 ----------- 31 May 2013 - CronJob ran 1476/1477 ----------- 30 May 2013 - CronJob ran 1475/1476 ----------- 29 May 2013 - CronJob ran 1474/1475 ----------- 28 May 2013 - CronJob ran 1473/1474 ----------- 27 May 2013 - CronJob ran 1472/1473 ----------- 26 May 2013 - CronJob ran 1471/1472 ----------- 25 May 2013 - CronJob ran 1470/1471 ----------- 24 May 2013 - CronJob ran 1469/1470 ----------- 23 May 2013 - CronJob ran 1468/1469 ----------- 22 May 2013 - CronJob ran 1467/1468 ----------- 21 May 2013 - CronJob ran 1466/1467 ----------- 20 May 2013 - CronJob ran 1465/1466 ----------- 19 May 2013 - CronJob ran 1464/1465 ----------- 18 May 2013 - CronJob ran 1463/1464 ----------- 17 May 2013 - CronJob ran 1462/1463 ----------- 16 May 2013 - CronJob ran 1461/1462 ----------- 15 May 2013 - CronJob ran 1460/1461 ----------- 14 May 2013 - CronJob ran 1459/1460 ----------- 13 May 2013 - CronJob ran 1458/1459 ----------- 12 May 2013 - CronJob ran 1457/1458 ----------- 11 May 2013 - CronJob ran 1456/1457 ----------- 10 May 2013 - CronJob ran 1455/1456 ----------- 09 May 2013 - CronJob ran 1454/1455 ----------- 08 May 2013 - CronJob ran 1453/1454 ----------- 07 May 2013 - CronJob ran 1452/1453 ----------- 06 May 2013 - CronJob ran 1451/1452 ----------- 05 May 2013 - CronJob ran 1450/1451 ----------- 04 May 2013 - CronJob ran 1449/1450 ----------- 03 May 2013 - CronJob ran 1448/1449 ----------- 02 May 2013 - CronJob ran 1447/1448 ----------- 01 May 2013 - CronJob ran 1446/1447 ----------- 30 Apr 2013 - CronJob ran 1445/1446 ----------- 29 Apr 2013 - CronJob ran 1444/1445 ----------- 28 Apr 2013 - CronJob ran 1443/1444 ----------- 27 Apr 2013 - CronJob ran 1442/1443 ----------- 26 Apr 2013 - CronJob ran 1441/1442 ----------- 25 Apr 2013 - CronJob ran 1440/1441 ----------- 24 Apr 2013 - CronJob ran 1439/1440 ----------- 23 Apr 2013 - CronJob ran 1438/1439 ----------- 22 Apr 2013 - CronJob ran 1437/1438 ----------- 21 Apr 2013 - CronJob ran 1436/1437 ----------- 20 Apr 2013 - CronJob ran 1435/1436 ----------- 19 Apr 2013 - CronJob ran 1434/1435 ----------- 18 Apr 2013 - CronJob ran 1433/1434 ----------- 17 Apr 2013 - CronJob ran 1432/1433 ----------- 16 Apr 2013 - CronJob ran 1431/1432 ----------- 15 Apr 2013 - CronJob ran 1430/1431 ----------- 14 Apr 2013 - CronJob ran 1429/1430 ----------- 13 Apr 2013 - CronJob ran 1428/1429 ----------- 12 Apr 2013 - CronJob ran 1427/1428 ----------- 11 Apr 2013 - CronJob ran 1426/1427 ----------- 10 Apr 2013 - CronJob ran 1425/1426 ----------- 09 Apr 2013 - CronJob ran 1424/1425 ----------- 08 Apr 2013 - CronJob ran 1423/1424 ----------- 07 Apr 2013 - CronJob ran 1422/1423 ----------- 06 Apr 2013 - CronJob ran 1421/1422 ----------- 05 Apr 2013 - CronJob ran 1420/1421 ----------- 04 Apr 2013 - CronJob ran 1419/1420 ----------- 03 Apr 2013 - CronJob ran 1418/1419 ----------- 02 Apr 2013 - CronJob ran 1417/1418 ----------- 01 Apr 2013 - CronJob ran 1416/1417 ----------- 31 Mar 2013 - CronJob ran 1415/1416 ----------- 30 Mar 2013 - CronJob ran 1414/1415 ----------- 29 Mar 2013 - CronJob ran 1413/1414 ----------- 28 Mar 2013 - CronJob ran 1412/1413 ----------- 27 Mar 2013 - CronJob ran 1411/1412 ----------- 26 Mar 2013 - CronJob ran 1410/1411 ----------- 25 Mar 2013 - CronJob ran 1409/1410 ----------- 24 Mar 2013 - CronJob ran 1408/1409 ----------- 23 Mar 2013 - CronJob ran 1407/1408 ----------- 22 Mar 2013 - CronJob ran 1406/1407 ----------- 21 Mar 2013 - CronJob ran 1405/1406 ----------- 20 Mar 2013 - CronJob ran 1404/1405 ----------- 19 Mar 2013 - CronJob ran 1403/1404 ----------- 18 Mar 2013 - CronJob ran 1402/1403 ----------- 17 Mar 2013 - CronJob ran 1401/1402 ----------- 16 Mar 2013 - CronJob ran 1400/1401 ----------- 15 Mar 2013 - CronJob ran 1399/1400 ----------- 14 Mar 2013 - CronJob ran 1398/1399 ----------- 13 Mar 2013 - CronJob ran 1397/1398 ----------- 12 Mar 2013 - CronJob ran 1396/1397 ----------- 11 Mar 2013 - CronJob ran 1395/1396 ----------- 10 Mar 2013 - CronJob ran 1394/1395 ----------- 09 Mar 2013 - CronJob ran 1393/1394 ----------- 08 Mar 2013 - CronJob ran 1392/1393 ----------- 07 Mar 2013 - CronJob ran 1391/1392 ----------- 06 Mar 2013 - CronJob ran 1390/1391 ----------- 05 Mar 2013 - CronJob ran 1389/1390 ----------- 04 Mar 2013 - CronJob ran 1388/1389 ----------- 03 Mar 2013 - CronJob ran 1387/1388 ----------- 02 Mar 2013 - CronJob ran 1386/1387 ----------- 01 Mar 2013 - CronJob ran 1385/1386 ----------- 28 Feb 2013 - CronJob ran 1384/1385 ----------- 27 Feb 2013 - CronJob ran 1383/1384 ----------- 26 Feb 2013 - CronJob ran 1382/1383 ----------- 25 Feb 2013 - CronJob ran 1381/1382 ----------- 24 Feb 2013 - CronJob ran 1380/1381 ----------- 23 Feb 2013 - CronJob ran 1379/1380 ----------- 22 Feb 2013 - CronJob ran 1378/1379 ----------- 21 Feb 2013 - CronJob ran 1377/1378 ----------- 20 Feb 2013 - CronJob ran 1376/1377 ----------- 19 Feb 2013 - CronJob ran 1375/1376 ----------- 18 Feb 2013 - CronJob ran 1374/1375 ----------- 17 Feb 2013 - CronJob ran 1373/1374 ----------- 16 Feb 2013 - CronJob ran 1372/1373 ----------- 15 Feb 2013 - CronJob ran 1371/1372 ----------- 14 Feb 2013 - CronJob ran 1370/1371 ----------- 13 Feb 2013 - CronJob ran 1369/1370 ----------- 12 Feb 2013 - CronJob ran 1368/1369 ----------- 11 Feb 2013 - CronJob ran 1367/1368 ----------- 10 Feb 2013 - CronJob ran 1366/1367 ----------- 09 Feb 2013 - CronJob ran 1365/1366 ----------- 08 Feb 2013 - CronJob ran 1364/1365 ----------- 07 Feb 2013 - CronJob ran 1363/1364 ----------- 06 Feb 2013 - CronJob ran 1362/1363 ----------- 05 Feb 2013 - CronJob ran 1361/1362 ----------- 04 Feb 2013 - CronJob ran 1360/1361 ----------- 03 Feb 2013 - CronJob ran 1359/1360 ----------- 02 Feb 2013 - CronJob ran 1358/1359 ----------- 01 Feb 2013 - CronJob ran 1357/1358 ----------- 31 Jan 2013 - CronJob ran 1356/1357 ----------- 30 Jan 2013 - CronJob ran 1355/1356 ----------- 29 Jan 2013 - CronJob ran 1354/1355 ----------- 28 Jan 2013 - CronJob ran 1353/1354 ----------- 27 Jan 2013 - CronJob ran 1352/1353 ----------- 26 Jan 2013 - CronJob ran 1351/1352 ----------- 25 Jan 2013 - CronJob ran 1350/1351 ----------- 24 Jan 2013 - CronJob ran 1349/1350 ----------- 23 Jan 2013 - CronJob ran 1348/1349 ----------- 22 Jan 2013 - CronJob ran 1347/1348 ----------- 21 Jan 2013 - CronJob ran 1346/1347 ----------- 20 Jan 2013 - CronJob ran 1345/1346 ----------- 19 Jan 2013 - CronJob ran 1344/1345 ----------- 18 Jan 2013 - CronJob ran 1343/1344 ----------- 17 Jan 2013 - CronJob ran 1342/1343 ----------- 16 Jan 2013 - CronJob ran 1341/1342 ----------- 15 Jan 2013 - CronJob ran 1340/1341 ----------- 14 Jan 2013 - CronJob ran 1339/1340 ----------- 13 Jan 2013 - CronJob ran 1338/1339 ----------- 12 Jan 2013 - CronJob ran 1337/1338 ----------- 11 Jan 2013 - CronJob ran 1336/1337 ----------- 10 Jan 2013 - CronJob ran 1335/1336 ----------- 09 Jan 2013 - CronJob ran 1334/1335 ----------- 08 Jan 2013 - CronJob ran 1333/1334 ----------- 07 Jan 2013 - CronJob ran 1332/1333 ----------- 06 Jan 2013 - CronJob ran 1331/1332 ----------- 05 Jan 2013 - CronJob ran 1330/1331 ----------- 04 Jan 2013 - CronJob ran 1329/1330 ----------- 03 Jan 2013 - CronJob ran 1328/1329 ----------- 02 Jan 2013 - CronJob ran 1327/1328 ----------- 01 Jan 2013 - CronJob ran 1326/1327 ----------- 31 Dec 2012 - CronJob ran 1325/1326 ----------- 30 Dec 2012 - CronJob ran 1324/1325 ----------- 29 Dec 2012 - CronJob ran 1323/1324 ----------- 28 Dec 2012 - CronJob ran 1323/1324 ----------- 28 Dec 2012 - CronJob ran 1322/1323 ----------- 28 Dec 2012 - CronJob ran 1321/1322 ----------- 27 Dec 2012 - CronJob ran 1320/1321 ----------- 26 Dec 2012 - CronJob ran 1319/1320 ----------- 25 Dec 2012 - CronJob ran 1318/1319 ----------- 24 Dec 2012 - CronJob ran 1317/1318 ----------- 23 Dec 2012 - CronJob ran 1316/1317 ----------- 22 Dec 2012 - CronJob ran 1315/1316 ----------- 21 Dec 2012 - CronJob ran 1314/1315 ----------- 20 Dec 2012 - CronJob ran 1313/1314 ----------- 19 Dec 2012 - CronJob ran 1312/1313 ----------- 17 Dec 2012 - CronJob ran 1311/1312 ----------- 16 Dec 2012 - CronJob ran 1310/1311 ----------- 15 Dec 2012 - CronJob ran 1309/1310 ----------- 14 Dec 2012 - CronJob ran 1308/1309 ----------- 13 Dec 2012 - CronJob ran 1307/1308 ----------- 12 Dec 2012 - CronJob ran 1306/1307 ----------- 11 Dec 2012 - CronJob ran 1305/1306 ----------- 11 Dec 2012 - CronJob ran 1304/1305 ----------- 10 Dec 2012 - CronJob ran 1303/1304 ----------- 09 Dec 2012 - CronJob ran 1302/1303 ----------- 08 Dec 2012 - CronJob ran 1301/1302 ----------- 07 Dec 2012 - CronJob ran 1300/1301 ----------- 06 Dec 2012 - CronJob ran 1300/1301 ----------- 06 Dec 2012 - CronJob ran 1299/1300 ----------- 05 Dec 2012 - CronJob ran 1298/1299 ----------- 05 Dec 2012 - CronJob ran 1297/1298 ----------- 05 Dec 2012 - CronJob ran 1296/1297 ----------- 05 Dec 2012 - CronJob ran 1295/1296 ----------- 05 Dec 2012 - CronJob ran 1294/1295 ----------- 29 Nov 2012 - CronJob ran 1293/1294 ----------- 28 Nov 2012 - CronJob ran 1292/1293 ----------- 27 Nov 2012 - CronJob ran 1291/1292 ----------- 26 Nov 2012 - CronJob ran 1290/1291 ----------- 26 Nov 2012 - CronJob ran 1289/1290 ----------- 26 Nov 2012 - CronJob ran 1288/1289 ----------- 23 Nov 2012 - CronJob ran 1287/1288 ----------- 22 Nov 2012 - CronJob ran 1286/1287 ----------- 21 Nov 2012 - CronJob ran 1285/1286 ----------- 21 Nov 2012 - CronJob ran 1284/1285 ----------- 19 Nov 2012 - CronJob ran 1283/1284 ----------- 19 Nov 2012 - CronJob ran 1282/1283 ----------- 19 Nov 2012 - CronJob ran 1281/1282 ----------- 16 Nov 2012 - CronJob ran 1280/1281 ----------- 15 Nov 2012 - CronJob ran 1279/1280 ----------- 14 Nov 2012 - CronJob ran 1278/1279 ----------- 13 Nov 2012 - CronJob ran 1277/1278 ----------- 12 Nov 2012 - CronJob ran 1276/1277 ----------- 12 Nov 2012 - CronJob ran 1275/1276 ----------- 10 Nov 2012 - CronJob ran 1274/1275 ----------- 09 Nov 2012 - CronJob ran 1273/1274 ----------- 09 Nov 2012 - CronJob ran 1272/1273 ----------- 07 Nov 2012 - CronJob ran 1271/1272 ----------- 06 Nov 2012 - CronJob ran 1270/1271 ----------- 05 Nov 2012 - CronJob ran 1269/1270 ----------- 04 Nov 2012 - CronJob ran 1268/1269 ----------- 03 Nov 2012 - CronJob ran 1267/1268 ----------- 02 Nov 2012 - CronJob ran 1266/1267 ----------- 01 Nov 2012 - CronJob ran 1265/1266 ----------- 31 Oct 2012 - CronJob ran 1264/1265 ----------- 30 Oct 2012 - CronJob ran 1263/1264 ----------- 29 Oct 2012 - CronJob ran 1262/1263 ----------- 28 Oct 2012 - CronJob ran 1261/1262 ----------- 27 Oct 2012 - CronJob ran 1260/1261 ----------- 26 Oct 2012 - CronJob ran 1259/1260 ----------- 25 Oct 2012 - CronJob ran 1258/1259 ----------- 24 Oct 2012 - CronJob ran 1257/1258 ----------- 23 Oct 2012 - CronJob ran 1256/1257 ----------- 22 Oct 2012 - CronJob ran 1255/1256 ----------- 21 Oct 2012 - CronJob ran 1254/1255 ----------- 20 Oct 2012 - CronJob ran 1253/1254 ----------- 19 Oct 2012 - CronJob ran 1252/1253 ----------- 18 Oct 2012 - CronJob ran 1251/1252 ----------- 17 Oct 2012 - CronJob ran 1250/1251 ----------- 16 Oct 2012 - CronJob ran 1249/1250 ----------- 15 Oct 2012 - CronJob ran 1248/1249 ----------- 14 Oct 2012 - CronJob ran 1247/1248 ----------- 13 Oct 2012 - CronJob ran 1246/1247 ----------- 12 Oct 2012 - CronJob ran 1245/1246 ----------- 11 Oct 2012 - CronJob ran 1244/1245 ----------- 10 Oct 2012 - CronJob ran 1243/1244 ----------- 09 Oct 2012 - CronJob ran 1242/1243 ----------- 08 Oct 2012 - CronJob ran 1241/1242 ----------- 07 Oct 2012 - CronJob ran 1240/1241 ----------- 06 Oct 2012 - CronJob ran 1239/1240 ----------- 05 Oct 2012 - CronJob ran 1238/1239 ----------- 04 Oct 2012 - CronJob ran 1237/1238 ----------- 03 Oct 2012 - CronJob ran 1236/1237 ----------- 02 Oct 2012 - CronJob ran 1235/1236 ----------- 01 Oct 2012 - CronJob ran 1234/1235 ----------- 30 Sep 2012 - CronJob ran 1233/1234 ----------- 29 Sep 2012 - CronJob ran 1232/1233 ----------- 28 Sep 2012 - CronJob ran 1231/1232 ----------- 27 Sep 2012 - CronJob ran 1230/1231 ----------- 26 Sep 2012 - CronJob ran 1229/1230 ----------- 25 Sep 2012 - CronJob ran 1228/1229 ----------- 24 Sep 2012 - CronJob ran 1227/1228 ----------- 23 Sep 2012 - CronJob ran 1226/1227 ----------- 22 Sep 2012 - CronJob ran 1225/1226 ----------- 21 Sep 2012 - CronJob ran 1224/1225 ----------- 20 Sep 2012 - CronJob ran 1223/1224 ----------- 19 Sep 2012 - CronJob ran 1222/1223 ----------- 18 Sep 2012 - CronJob ran 1221/1222 ----------- 17 Sep 2012 - CronJob ran 1220/1221 ----------- 16 Sep 2012 - CronJob ran 1219/1220 ----------- 15 Sep 2012 - CronJob ran 1218/1219 ----------- 14 Sep 2012 - CronJob ran 1217/1218 ----------- 13 Sep 2012 - CronJob ran 1216/1217 ----------- 12 Sep 2012 - CronJob ran 1215/1216 ----------- 11 Sep 2012 - CronJob ran 1214/1215 ----------- 10 Sep 2012 - CronJob ran 1213/1214 ----------- 09 Sep 2012 - CronJob ran 1212/1213 ----------- 08 Sep 2012 - CronJob ran 1211/1212 ----------- 07 Sep 2012 - CronJob ran 1210/1211 ----------- 06 Sep 2012 - CronJob ran 1209/1210 ----------- 05 Sep 2012 - CronJob ran 1208/1209 ----------- 04 Sep 2012 - CronJob ran 1207/1208 ----------- 03 Sep 2012 - CronJob ran 1206/1207 ----------- 02 Sep 2012 - CronJob ran 1205/1206 ----------- 01 Sep 2012 - CronJob ran 1204/1205 ----------- 31 Aug 2012 - CronJob ran 1203/1204 ----------- 30 Aug 2012 - CronJob ran 1202/1203 ----------- 29 Aug 2012 - CronJob ran 1201/1202 ----------- 28 Aug 2012 - CronJob ran 1200/1201 ----------- 27 Aug 2012 - CronJob ran 1199/1200 ----------- 26 Aug 2012 - CronJob ran 1198/1199 ----------- 25 Aug 2012 - CronJob ran 1197/1198 ----------- 25 Aug 2012 - CronJob ran 1199/1200 ----------- 24 Aug 2012 - CronJob ran 1198/1199 ----------- 24 Aug 2012 - CronJob ran 1198/1199 ----------- 23 Aug 2012 - CronJob ran 1196/1197 ----------- 23 Aug 2012 - CronJob ran 1196/1197 ----------- 22 Aug 2012 - CronJob ran 1195/1196 ----------- 22 Aug 2012 - CronJob ran 1195/1196 ----------- 21 Aug 2012 - CronJob ran 1194/1195 ----------- 21 Aug 2012 - CronJob ran 1194/1195 ----------- 20 Aug 2012 - CronJob ran 1193/1194 ----------- 20 Aug 2012 - CronJob ran 1193/1194 ----------- 19 Aug 2012 - CronJob ran 1191/1192 ----------- 19 Aug 2012 - CronJob ran 1191/1192 ----------- 18 Aug 2012 - CronJob ran 1190/1191 ----------- 18 Aug 2012 - CronJob ran 1190/1191 ----------- 17 Aug 2012 - CronJob ran 1189/1190 ----------- 17 Aug 2012 - CronJob ran 1189/1190 ----------- 16 Aug 2012 - CronJob ran 1188/1189 ----------- 16 Aug 2012 - CronJob ran 1188/1189 ----------- 15 Aug 2012 - CronJob ran 1187/1188 ----------- 14 Aug 2012 - CronJob ran 1186/1187 ----------- 13 Aug 2012 - CronJob ran 1185/1186 ----------- 12 Aug 2012 - CronJob ran 1184/1185 ----------- 11 Aug 2012 - CronJob ran 1183/1184 ----------- 10 Aug 2012 - CronJob ran 1182/1183 ----------- 09 Aug 2012 - CronJob ran 1181/1182 ----------- 08 Aug 2012 - CronJob ran 1180/1181 ----------- 07 Aug 2012 - CronJob ran 1179/1180 ----------- 06 Aug 2012 - CronJob ran 1178/1179 ----------- 05 Aug 2012 - CronJob ran 1177/1178 ----------- 04 Aug 2012 - CronJob ran 1176/1177 ----------- 03 Aug 2012 - CronJob ran 1175/1176 ----------- 02 Aug 2012 - CronJob ran 1174/1175 ----------- 01 Aug 2012 - CronJob ran 1173/1174 ----------- 31 Jul 2012 - CronJob ran 1172/1173 ----------- 30 Jul 2012 - CronJob ran 1171/1172 ----------- 29 Jul 2012 - CronJob ran 1170/1171 ----------- 28 Jul 2012 - CronJob ran 1169/1170 ----------- 27 Jul 2012 - CronJob ran 1168/1169 ----------- 26 Jul 2012 - CronJob ran 1167/1168 ----------- 25 Jul 2012 - CronJob ran 1166/1167 ----------- 24 Jul 2012 - CronJob ran 1165/1166 ----------- 23 Jul 2012 - CronJob ran 1164/1165 ----------- 22 Jul 2012 - CronJob ran 1163/1164 ----------- 21 Jul 2012 - CronJob ran 1162/1163 ----------- 20 Jul 2012 - CronJob ran 1161/1162 ----------- 19 Jul 2012 - CronJob ran 1160/1161 ----------- 18 Jul 2012 - CronJob ran 1159/1160 ----------- 17 Jul 2012 - CronJob ran 1158/1159 ----------- 16 Jul 2012 - CronJob ran 1157/1158 ----------- 15 Jul 2012 - CronJob ran 1156/1157 ----------- 14 Jul 2012 - CronJob ran 1155/1156 ----------- 13 Jul 2012 - CronJob ran 1154/1155 ----------- 12 Jul 2012 - CronJob ran 1153/1154 ----------- 11 Jul 2012 - CronJob ran 1152/1153 ----------- 10 Jul 2012 - CronJob ran 1151/1152 ----------- 09 Jul 2012 - CronJob ran 1150/1151 ----------- 08 Jul 2012 - CronJob ran 1149/1150 ----------- 07 Jul 2012 - CronJob ran 1148/1149 ----------- 06 Jul 2012 - CronJob ran 1147/1148 ----------- 05 Jul 2012 - CronJob ran 1146/1147 ----------- 04 Jul 2012 - CronJob ran 1145/1146 ----------- 03 Jul 2012 - CronJob ran 1144/1145 ----------- 02 Jul 2012 - CronJob ran 1143/1144 ----------- 01 Jul 2012 - CronJob ran 1142/1143 ----------- 30 Jun 2012 - CronJob ran 1141/1142 ----------- 29 Jun 2012 - CronJob ran 1140/1141 ----------- 28 Jun 2012 - CronJob ran 1139/1140 ----------- 27 Jun 2012 - CronJob ran 1138/1139 ----------- 26 Jun 2012 - CronJob ran 1137/1138 ----------- 25 Jun 2012 - CronJob ran 1136/1137 ----------- 24 Jun 2012 - CronJob ran 1135/1136 ----------- 23 Jun 2012 - CronJob ran 1134/1135 ----------- 22 Jun 2012 - CronJob ran 1133/1134 ----------- 21 Jun 2012 - CronJob ran 1132/1133 ----------- 20 Jun 2012 - CronJob ran 1131/1132 ----------- 19 Jun 2012 - CronJob ran 1130/1131 ----------- 18 Jun 2012 - CronJob ran 1129/1130 ----------- 17 Jun 2012 - CronJob ran 1128/1129 ----------- 16 Jun 2012 - CronJob ran 1127/1128 ----------- 15 Jun 2012 - CronJob ran 1126/1127 ----------- 14 Jun 2012 - CronJob ran 1125/1126 ----------- 13 Jun 2012 - CronJob ran 1124/1125 ----------- 12 Jun 2012 - CronJob ran 1123/1124 ----------- 11 Jun 2012 - CronJob ran 1122/1123 ----------- 10 Jun 2012 - CronJob ran 1121/1122 ----------- 09 Jun 2012 - CronJob ran 1120/1121 ----------- 08 Jun 2012 - CronJob ran 1119/1120 ----------- 07 Jun 2012 - CronJob ran 1118/1119 ----------- 06 Jun 2012 - CronJob ran 1117/1118 ----------- 05 Jun 2012 - CronJob ran 1116/1117 ----------- 04 Jun 2012 - CronJob ran 1115/1116 ----------- 03 Jun 2012 - CronJob ran 1114/1115 ----------- 02 Jun 2012 - CronJob ran 1113/1114 ----------- 01 Jun 2012 - CronJob ran 1112/1113 ----------- 31 May 2012 - CronJob ran 1111/1112 ----------- 30 May 2012 - CronJob ran 1110/1111 ----------- 29 May 2012 - CronJob ran 1109/1110 ----------- 28 May 2012 - CronJob ran 1108/1109 ----------- 27 May 2012 - CronJob ran 1107/1108 ----------- 26 May 2012 - CronJob ran 1106/1107 ----------- 25 May 2012 - CronJob ran 1105/1106 ----------- 24 May 2012 - CronJob ran 1104/1105 ----------- 23 May 2012 - CronJob ran 1103/1104 ----------- 22 May 2012 - CronJob ran 1102/1103 ----------- 21 May 2012 - CronJob ran 1101/1102 ----------- 20 May 2012 - CronJob ran 1100/1101 ----------- 19 May 2012 - CronJob ran 1099/1100 ----------- 18 May 2012 - CronJob ran 1098/1099 ----------- 17 May 2012 - CronJob ran 1097/1098 ----------- 16 May 2012 - CronJob ran 1096/1097 ----------- 15 May 2012 - CronJob ran 1095/1096 ----------- 14 May 2012 - CronJob ran 1094/1095 ----------- 13 May 2012 - CronJob ran 1093/1094 ----------- 12 May 2012 - CronJob ran 1092/1093 ----------- 11 May 2012 - CronJob ran 1091/1092 ----------- 10 May 2012 - CronJob ran 1090/1091 ----------- 09 May 2012 - CronJob ran 1089/1090 ----------- 08 May 2012 - CronJob ran 1088/1089 ----------- 07 May 2012 - CronJob ran 1087/1088 ----------- 06 May 2012 - CronJob ran 1086/1087 ----------- 05 May 2012 - CronJob ran 1085/1086 ----------- 04 May 2012 - CronJob ran 1084/1085 ----------- 03 May 2012 - CronJob ran 1083/1084 ----------- 02 May 2012 - CronJob ran 1082/1083 ----------- 01 May 2012 - CronJob ran 1081/1082 ----------- 30 Apr 2012 - CronJob ran 1080/1081 ----------- 29 Apr 2012 - CronJob ran 1079/1080 ----------- 28 Apr 2012 - CronJob ran 1078/1079 ----------- 27 Apr 2012 - CronJob ran 1077/1078 ----------- 26 Apr 2012 - CronJob ran 1076/1077 ----------- 25 Apr 2012 - CronJob ran 1075/1076 ----------- 24 Apr 2012 - CronJob ran 1074/1075 ----------- 23 Apr 2012 - CronJob ran 1073/1074 ----------- 22 Apr 2012 - CronJob ran 1072/1073 ----------- 21 Apr 2012 - CronJob ran 1071/1072 ----------- 20 Apr 2012 - CronJob ran 1070/1071 ----------- 19 Apr 2012 - CronJob ran 1069/1070 ----------- 18 Apr 2012 - CronJob ran 1068/1069 ----------- 17 Apr 2012 - CronJob ran 1067/1068 ----------- 16 Apr 2012 - CronJob ran 1066/1067 ----------- 15 Apr 2012 - CronJob ran 1065/1066 ----------- 14 Apr 2012 - CronJob ran 1064/1065 ----------- 13 Apr 2012 - CronJob ran 1063/1064 ----------- 12 Apr 2012 - CronJob ran 1062/1063 ----------- 11 Apr 2012 - CronJob ran 1061/1062 ----------- 10 Apr 2012 - CronJob ran 1060/1061 ----------- 09 Apr 2012 - CronJob ran 1059/1060 ----------- 08 Apr 2012 - CronJob ran 1058/1059 ----------- 07 Apr 2012 - CronJob ran 1057/1058 ----------- 06 Apr 2012 - CronJob ran 1056/1057 ----------- 05 Apr 2012 - CronJob ran 1055/1056 ----------- 04 Apr 2012 - CronJob ran 1054/1055 ----------- 03 Apr 2012 - CronJob ran 1053/1054 ----------- 02 Apr 2012 - CronJob ran 1052/1053 ----------- 01 Apr 2012 - CronJob ran 1051/1052 ----------- 31 Mar 2012 - CronJob ran 1050/1051 ----------- 30 Mar 2012 - CronJob ran 1049/1050 ----------- 29 Mar 2012 - CronJob ran 1048/1049 ----------- 28 Mar 2012 - CronJob ran 1047/1048 ----------- 27 Mar 2012 - CronJob ran 1046/1047 ----------- 26 Mar 2012 - CronJob ran 1045/1046 ----------- 25 Mar 2012 - CronJob ran 1044/1045 ----------- 24 Mar 2012 - CronJob ran 1043/1044 ----------- 23 Mar 2012 - CronJob ran 1042/1043 ----------- 22 Mar 2012 - CronJob ran 1041/1042 ----------- 21 Mar 2012 - CronJob ran 1040/1041 ----------- 20 Mar 2012 - CronJob ran 1039/1040 ----------- 19 Mar 2012 - CronJob ran 1038/1039 ----------- 18 Mar 2012 - CronJob ran 1037/1038 ----------- 17 Mar 2012 - CronJob ran 1036/1037 ----------- 16 Mar 2012 - CronJob ran 1035/1036 ----------- 15 Mar 2012 - CronJob ran 1034/1035 ----------- 14 Mar 2012 - CronJob ran 1033/1034 ----------- 13 Mar 2012 - CronJob ran 1032/1033 ----------- 12 Mar 2012 - CronJob ran 1031/1032 ----------- 11 Mar 2012 - CronJob ran 1030/1031 ----------- 10 Mar 2012 - CronJob ran 1029/1030 ----------- 09 Mar 2012 - CronJob ran 1028/1029 ----------- 08 Mar 2012 - CronJob ran 1027/1028 ----------- 07 Mar 2012 - CronJob ran 1026/1027 ----------- 06 Mar 2012 - CronJob ran 1025/1026 ----------- 05 Mar 2012 - CronJob ran 1024/1025 ----------- 04 Mar 2012 - CronJob ran 1023/1024 ----------- 03 Mar 2012 - CronJob ran 1022/1023 ----------- 02 Mar 2012 - CronJob ran 1021/1022 ----------- 01 Mar 2012 - CronJob ran 1020/1021 ----------- 29 Feb 2012 - CronJob ran 1019/1020 ----------- 28 Feb 2012 - CronJob ran 1018/1019 ----------- 27 Feb 2012 - CronJob ran 1017/1018 ----------- 26 Feb 2012 - CronJob ran 1016/1017 ----------- 25 Feb 2012 - CronJob ran 1015/1016 ----------- 24 Feb 2012 - CronJob ran 1014/1015 ----------- 23 Feb 2012 - CronJob ran 1013/1014 ----------- 22 Feb 2012 - CronJob ran 1012/1013 ----------- 21 Feb 2012 - CronJob ran 1011/1012 ----------- 20 Feb 2012 - CronJob ran 1010/1011 ----------- 19 Feb 2012 - CronJob ran 1009/1010 ----------- 18 Feb 2012 - CronJob ran 1008/1009 ----------- 17 Feb 2012 - CronJob ran 1007/1008 ----------- 16 Feb 2012 - CronJob ran 1006/1007 ----------- 15 Feb 2012 - CronJob ran 1005/1006 ----------- 14 Feb 2012 - CronJob ran 1004/1005 ----------- 13 Feb 2012 - CronJob ran 1003/1004 ----------- 12 Feb 2012 - CronJob ran 1002/1003 ----------- 11 Feb 2012 - CronJob ran 1001/1002 ----------- 10 Feb 2012 - CronJob ran 1000/1001 ----------- 09 Feb 2012 - CronJob ran 999/1000 ----------- 08 Feb 2012 - CronJob ran 998/999 ----------- 07 Feb 2012 - CronJob ran 997/998 ----------- 06 Feb 2012 - CronJob ran 996/997 ----------- 05 Feb 2012 - CronJob ran 995/996 ----------- 04 Feb 2012 - CronJob ran 994/995 ----------- 03 Feb 2012 - CronJob ran 993/994 ----------- 02 Feb 2012 - CronJob ran 992/993 ----------- 01 Feb 2012 - CronJob ran 991/992 ----------- 31 Jan 2012 - CronJob ran 990/991 ----------- 30 Jan 2012 - CronJob ran 989/990 ----------- 29 Jan 2012 - CronJob ran 988/989 ----------- 28 Jan 2012 - CronJob ran 987/988 ----------- 27 Jan 2012 - CronJob ran 986/987 ----------- 26 Jan 2012 - CronJob ran 985/986 ----------- 25 Jan 2012 - CronJob ran 984/985 ----------- 24 Jan 2012 - CronJob ran 983/984 ----------- 23 Jan 2012 - CronJob ran 982/983 ----------- 22 Jan 2012 - CronJob ran 981/982 ----------- 21 Jan 2012 - CronJob ran 980/981 ----------- 20 Jan 2012 - CronJob ran 979/980 ----------- 19 Jan 2012 - CronJob ran 978/979 ----------- 18 Jan 2012 - CronJob ran 977/978 ----------- 17 Jan 2012 - CronJob ran 976/977 ----------- 16 Jan 2012 - CronJob ran 975/976 ----------- 15 Jan 2012 - CronJob ran 974/975 ----------- 14 Jan 2012 - CronJob ran 973/974 ----------- 13 Jan 2012 - CronJob ran 972/973 ----------- 12 Jan 2012 - CronJob ran 971/972 ----------- 11 Jan 2012 - CronJob ran 970/971 ----------- 10 Jan 2012 - CronJob ran 969/970 ----------- 09 Jan 2012 - CronJob ran 968/969 ----------- 08 Jan 2012 - CronJob ran 967/968 ----------- 07 Jan 2012 - CronJob ran 966/967 ----------- 06 Jan 2012 - CronJob ran 965/966 ----------- 05 Jan 2012 - CronJob ran 964/965 ----------- 05 Jan 2012 - CronJob ran 963/964 ----------- 05 Jan 2012 - CronJob ran 962/963 ----------- 05 Jan 2012 - CronJob ran 961/962 ----------- 29 Dec 2011 - CronJob ran 957/958 ----------- 28 Dec 2011 - CronJob ran 956/957 ----------- 27 Dec 2011 - CronJob ran 955/956 ----------- 26 Dec 2011 - CronJob ran 954/955 ----------- 25 Dec 2011 - CronJob ran 953/954 ----------- 24 Dec 2011 - CronJob ran 952/953 ----------- 23 Dec 2011 - CronJob ran 951/952 ----------- 22 Dec 2011 - CronJob ran 950/951 ----------- 21 Dec 2011 - CronJob ran 949/950 ----------- 20 Dec 2011 - CronJob ran 948/949 ----------- 19 Dec 2011 - CronJob ran 947/948 ----------- 18 Dec 2011 - CronJob ran 946/947 ----------- 17 Dec 2011 - CronJob ran 945/946 ----------- 16 Dec 2011 - CronJob ran 944/945 ----------- 14 Dec 2011 - CronJob ran 942/943 ----------- 12 Dec 2011 - CronJob ran 940/941 ----------- 11 Dec 2011 - CronJob ran 939/940 ----------- 10 Dec 2011 - CronJob ran 938/939 ----------- 09 Dec 2011 - CronJob ran 937/938 ----------- 08 Dec 2011 - CronJob ran 936/937 ----------- 07 Dec 2011 - CronJob ran 935/936 ----------- 06 Dec 2011 - CronJob ran 934/935 ----------- 05 Dec 2011 - CronJob ran 933/934 ----------- 04 Dec 2011 - CronJob ran 932/933 ----------- 03 Dec 2011 - CronJob ran 931/932 ----------- 02 Dec 2011 - CronJob ran 930/931 ----------- 01 Dec 2011 - CronJob ran 929/930 ----------- 30 Nov 2011 - CronJob ran 928/929 ----------- 29 Nov 2011 - CronJob ran 927/928 ----------- 28 Nov 2011 - CronJob ran 926/927 ----------- 27 Nov 2011 - CronJob ran 925/926 ----------- 26 Nov 2011 - CronJob ran 924/925 ----------- 25 Nov 2011 - CronJob ran 923/924 ----------- 24 Nov 2011 - CronJob ran 922/923 ----------- 23 Nov 2011 - CronJob ran 921/922 ----------- 22 Nov 2011 - CronJob ran 920/921 ----------- 21 Nov 2011 - CronJob ran 919/920 ----------- 20 Nov 2011 - CronJob ran 918/919 ----------- 19 Nov 2011 - CronJob ran 917/918 ----------- 18 Nov 2011 - CronJob ran 916/917 ----------- 17 Nov 2011 - CronJob ran 915/916 ----------- 16 Nov 2011 - CronJob ran 914/915 ----------- 15 Nov 2011 - CronJob ran 913/914 ----------- 14 Nov 2011 - CronJob ran 912/913 ----------- 13 Nov 2011 - CronJob ran 911/912 ----------- 12 Nov 2011 - CronJob ran 910/911 ----------- 11 Nov 2011 - CronJob ran 909/910 ----------- 11 Nov 2011 - CronJob ran 910/911 ----------- 11 Nov 2011 - CronJob ran 909/910 ----------- 10 Nov 2011 - CronJob ran 908/909 ----------- 09 Nov 2011 - CronJob ran 907/908 ----------- 08 Nov 2011 - CronJob ran 906/907 ----------- 07 Nov 2011 - CronJob ran 905/906 ----------- 06 Nov 2011 - CronJob ran 904/905 ----------- 05 Nov 2011 - CronJob ran 903/904 ----------- 04 Nov 2011 - CronJob ran 902/903 ----------- 03 Nov 2011 - CronJob ran 901/902 ----------- 02 Nov 2011 - CronJob ran 900/901 ----------- 01 Nov 2011 - CronJob ran 899/900 ----------- 31 Oct 2011 - CronJob ran 898/899 ----------- 30 Oct 2011 - CronJob ran 897/898 ----------- 29 Oct 2011 - CronJob ran 896/897 ----------- 28 Oct 2011 - CronJob ran 895/896 ----------- 27 Oct 2011 - CronJob ran 894/895 ----------- 26 Oct 2011 - CronJob ran 893/894 ----------- 25 Oct 2011 - CronJob ran 892/893 ----------- 24 Oct 2011 - CronJob ran 891/892 ----------- 23 Oct 2011 - CronJob ran 890/891 ----------- 22 Oct 2011 - CronJob ran 889/890 ----------- 21 Oct 2011 - CronJob ran 888/889 ----------- 20 Oct 2011 - CronJob ran 887/888 ----------- 19 Oct 2011 - CronJob ran 886/887 ----------- 18 Oct 2011 - CronJob ran 885/886 ----------- 17 Oct 2011 - CronJob ran 884/885 ----------- 16 Oct 2011 - CronJob ran 883/884 ----------- 15 Oct 2011 - CronJob ran 882/883 ----------- 14 Oct 2011 - CronJob ran 881/882 ----------- 13 Oct 2011 - CronJob ran 880/881 ----------- 12 Oct 2011 - CronJob ran 879/880 ----------- 11 Oct 2011 - CronJob ran 878/879 ----------- 10 Oct 2011 - CronJob ran 877/878 ----------- 09 Oct 2011 - CronJob ran 876/877 ----------- 08 Oct 2011 - CronJob ran 875/876 ----------- 07 Oct 2011 - CronJob ran 874/875 ----------- 06 Oct 2011 - CronJob ran 873/874 ----------- 05 Oct 2011 - CronJob ran 872/873 ----------- 04 Oct 2011 - CronJob ran 871/872 ----------- 03 Oct 2011 - CronJob ran 870/871 ----------- 02 Oct 2011 - CronJob ran 869/870 ----------- 01 Oct 2011 - CronJob ran 868/869 ----------- 30 Sep 2011 - CronJob ran 867/868 ----------- 29 Sep 2011 - CronJob ran 866/867 ----------- 28 Sep 2011 - CronJob ran 865/866 ----------- 27 Sep 2011 - CronJob ran 864/865 ----------- 26 Sep 2011 - CronJob ran 863/864 ----------- 25 Sep 2011 - CronJob ran 862/863 ----------- 24 Sep 2011 - CronJob ran 861/862 ----------- 23 Sep 2011 - CronJob ran 860/861 ----------- 22 Sep 2011 - CronJob ran 859/860 ----------- 21 Sep 2011 - CronJob ran 858/859 ----------- 20 Sep 2011 - CronJob ran 857/858 ----------- 19 Sep 2011 - CronJob ran 856/857 ----------- 18 Sep 2011 - CronJob ran 855/856 ----------- 17 Sep 2011 - CronJob ran 854/855 ----------- 16 Sep 2011 - CronJob ran 853/854 ----------- 15 Sep 2011 - CronJob ran 852/853 ----------- 14 Sep 2011 - CronJob ran 851/852 ----------- 13 Sep 2011 - CronJob ran 850/851 ----------- 12 Sep 2011 - CronJob ran 849/850 ----------- 11 Sep 2011 - CronJob ran 848/849 ----------- 10 Sep 2011 - CronJob ran 847/848 ----------- 09 Sep 2011 - CronJob ran 846/847 ----------- 08 Sep 2011 - CronJob ran 845/846 ----------- 07 Sep 2011 - CronJob ran 844/845 ----------- 06 Sep 2011 - CronJob ran 843/844 ----------- 05 Sep 2011 - CronJob ran 842/843 ----------- 04 Sep 2011 - CronJob ran 841/842 ----------- 03 Sep 2011 - CronJob ran 840/841 ----------- 02 Sep 2011 - CronJob ran 839/840 ----------- 01 Sep 2011 - CronJob ran 838/839 ----------- 31 Aug 2011 - CronJob ran 837/838 ----------- 30 Aug 2011 - CronJob ran 836/837 ----------- 29 Aug 2011 - CronJob ran 835/836 ----------- 28 Aug 2011 - CronJob ran 834/835 ----------- 27 Aug 2011 - CronJob ran 833/834 ----------- 26 Aug 2011 - CronJob ran 832/833 ----------- 25 Aug 2011 - CronJob ran 831/832 ----------- 24 Aug 2011 - CronJob ran 830/831 ----------- 23 Aug 2011 - CronJob ran 829/830 ----------- 22 Aug 2011 - CronJob ran 828/829 ----------- 21 Aug 2011 - CronJob ran 827/828 ----------- 20 Aug 2011 - CronJob ran 826/827 ----------- 19 Aug 2011 - CronJob ran 825/826 ----------- 18 Aug 2011 - CronJob ran 824/825 ----------- 17 Aug 2011 - CronJob ran 823/824 ----------- 16 Aug 2011 - CronJob ran 822/823 ----------- 15 Aug 2011 - CronJob ran 821/822 ----------- 14 Aug 2011 - CronJob ran 820/821 ----------- 13 Aug 2011 - CronJob ran 819/820 ----------- 12 Aug 2011 - CronJob ran 818/819 ----------- 11 Aug 2011 - CronJob ran 817/818 ----------- 10 Aug 2011 - CronJob ran 816/817 ----------- 09 Aug 2011 - CronJob ran 815/816 ----------- 08 Aug 2011 - CronJob ran 814/815 ----------- 07 Aug 2011 - CronJob ran 813/814 ----------- 06 Aug 2011 - CronJob ran 812/813 ----------- 05 Aug 2011 - CronJob ran 811/812 ----------- 04 Aug 2011 - CronJob ran 810/811 ----------- 03 Aug 2011 - CronJob ran 809/810 ----------- 02 Aug 2011 - CronJob ran 808/809 ----------- 01 Aug 2011 - CronJob ran 807/808 ----------- 31 Jul 2011 - CronJob ran 806/807 ----------- 30 Jul 2011 - CronJob ran 805/806 ----------- 29 Jul 2011 - CronJob ran 804/805 ----------- 29 Jul 2011 - CronJob ran 803/804 ----------- 29 Jul 2011 - CronJob ran 804/805 ----------- 28 Jul 2011 - CronJob ran 803/804 ----------- 28 Jul 2011 - CronJob ran 803/804 ----------- 27 Jul 2011 - CronJob ran 802/803 ----------- 27 Jul 2011 - CronJob ran 802/803 ----------- 26 Jul 2011 - CronJob ran 801/802 ----------- 25 Jul 2011 - CronJob ran 800/801 ----------- 24 Jul 2011 - CronJob ran 799/800 ----------- 23 Jul 2011 - CronJob ran 798/799 ----------- 22 Jul 2011 - CronJob ran 797/798 ----------- 21 Jul 2011 - CronJob ran 796/797 ----------- 20 Jul 2011 - CronJob ran 795/796 ----------- 19 Jul 2011 - CronJob ran 794/795 ----------- 18 Jul 2011 - CronJob ran 793/794 ----------- 17 Jul 2011 - CronJob ran 792/793 ----------- 16 Jul 2011 - CronJob ran 791/792 ----------- 15 Jul 2011 - CronJob ran 790/791 ----------- 14 Jul 2011 - CronJob ran 789/790 ----------- 13 Jul 2011 - CronJob ran 788/789 ----------- 12 Jul 2011 - CronJob ran 787/788 ----------- 11 Jul 2011 - CronJob ran 786/787 ----------- 10 Jul 2011 - CronJob ran 785/786 ----------- 09 Jul 2011 - CronJob ran 784/785 ----------- 09 Jul 2011 - CronJob ran 784/785 ----------- 08 Jul 2011 - CronJob ran 783/784 ----------- 07 Jul 2011 - CronJob ran 782/783 ----------- 06 Jul 2011 - CronJob ran 781/782 ----------- 05 Jul 2011 - CronJob ran 780/781 ----------- 04 Jul 2011 - CronJob ran 779/780 ----------- 03 Jul 2011 - CronJob ran 778/779 ----------- 02 Jul 2011 - CronJob ran 777/778 ----------- 01 Jul 2011 - CronJob ran 776/777 ----------- 30 Jun 2011 - CronJob ran 775/776 ----------- 29 Jun 2011 - CronJob ran 774/775 ----------- 29 Jun 2011 - CronJob ran 773/774 ----------- 29 Jun 2011 - CronJob ran 772/773 ----------- 29 Jun 2011 - CronJob ran 771/772 ----------- 28 Jun 2011 - CronJob ran 770/771 ----------- 24 Jun 2011 - CronJob ran 769/770 ----------- 23 Jun 2011 - CronJob ran 768/769 ----------- 22 Jun 2011 - CronJob ran 767/768 ----------- 22 Jun 2011 - CronJob ran 767/768 ----------- 21 Jun 2011 - CronJob ran 766/767 ----------- 21 Jun 2011 - CronJob ran 766/767 ----------- 20 Jun 2011 - CronJob ran 765/766 ----------- 20 Jun 2011 - CronJob ran 765/766 ----------- 19 Jun 2011 - CronJob ran 764/765 ----------- 18 Jun 2011 - CronJob ran 763/764 ----------- 17 Jun 2011 - CronJob ran 762/763 ----------- 16 Jun 2011 - CronJob ran 761/762 ----------- 15 Jun 2011 - CronJob ran 760/761 ----------- 14 Jun 2011 - CronJob ran 759/760 ----------- 13 Jun 2011 - CronJob ran 758/759 ----------- 12 Jun 2011 - CronJob ran 757/758 ----------- 11 Jun 2011 - CronJob ran 756/757 ----------- 10 Jun 2011 - CronJob ran 755/756 ----------- 09 Jun 2011 - CronJob ran 754/755 ----------- 08 Jun 2011 - CronJob ran 753/754 ----------- 07 Jun 2011 - CronJob ran 752/753 ----------- 06 Jun 2011 - CronJob ran 751/752 ----------- 05 Jun 2011 - CronJob ran 750/751 ----------- 04 Jun 2011 - CronJob ran 749/750 ----------- 03 Jun 2011 - CronJob ran 748/749 ----------- 02 Jun 2011 - CronJob ran 747/748 ----------- 01 Jun 2011 - CronJob ran 746/747 ----------- 31 May 2011 - CronJob ran 745/746 ----------- 30 May 2011 - CronJob ran 744/745 ----------- 29 May 2011 - CronJob ran 743/744 ----------- 28 May 2011 - CronJob ran 742/743 ----------- 27 May 2011 - CronJob ran 741/742 ----------- 27 May 2011 - CronJob ran 741/742 ----------- 26 May 2011 - CronJob ran 740/741 ----------- 25 May 2011 - CronJob ran 739/740 ----------- 25 May 2011 - CronJob ran 738/739 ----------- 25 May 2011 - CronJob ran 739/740 ----------- 24 May 2011 - CronJob ran 738/739 ----------- 23 May 2011 - CronJob ran 737/738 ----------- 22 May 2011 - CronJob ran 736/737 ----------- 21 May 2011 - CronJob ran 735/736 ----------- 20 May 2011 - CronJob ran 734/735 ----------- 19 May 2011 - CronJob ran 733/734 ----------- 18 May 2011 - CronJob ran 732/733 ----------- 17 May 2011 - CronJob ran 731/732 ----------- 16 May 2011 - CronJob ran 730/731 ----------- 15 May 2011 - CronJob ran 729/730 ----------- 14 May 2011 - CronJob ran 728/729 ----------- 13 May 2011 - CronJob ran 727/728 ----------- 12 May 2011 - CronJob ran 726/727 ----------- 11 May 2011 - CronJob ran 725/726 ----------- 10 May 2011 - CronJob ran 724/725 ----------- 09 May 2011 - CronJob ran 723/724 ----------- 08 May 2011 - CronJob ran 722/723 ----------- 07 May 2011 - CronJob ran 721/722 ----------- 06 May 2011 - CronJob ran 720/721 ----------- 05 May 2011 - CronJob ran 719/720 ----------- 04 May 2011 - CronJob ran 718/719 ----------- 03 May 2011 - CronJob ran 717/718 ----------- 02 May 2011 - CronJob ran 716/717 ----------- 01 May 2011 - CronJob ran 715/716 ----------- 30 Apr 2011 - CronJob ran 714/715 ----------- 29 Apr 2011 - CronJob ran 713/714 ----------- 28 Apr 2011 - CronJob ran 712/713 ----------- 26 Apr 2011 - CronJob ran 710/711 ----------- 25 Apr 2011 - CronJob ran 709/710 ----------- 24 Apr 2011 - CronJob ran 708/709 ----------- 23 Apr 2011 - CronJob ran 707/708 ----------- 22 Apr 2011 - CronJob ran 706/707 ----------- 21 Apr 2011 - CronJob ran 705/706 ----------- 20 Apr 2011 - CronJob ran 704/705 ----------- 19 Apr 2011 - CronJob ran 703/704 ----------- 18 Apr 2011 - CronJob ran 702/703 ----------- 17 Apr 2011 - CronJob ran 701/702 ----------- 16 Apr 2011 - CronJob ran 700/701 ----------- 15 Apr 2011 - CronJob ran 699/700 ----------- 14 Apr 2011 - CronJob ran 698/699 ----------- 13 Apr 2011 - CronJob ran 697/698 ----------- 12 Apr 2011 - CronJob ran 696/697 ----------- 11 Apr 2011 - CronJob ran 695/696 ----------- 10 Apr 2011 - CronJob ran 694/695 ----------- 09 Apr 2011 - CronJob ran 693/694 ----------- 08 Apr 2011 - CronJob ran 692/693 ----------- 07 Apr 2011 - CronJob ran 691/692 ----------- 06 Apr 2011 - CronJob ran 690/691 ----------- 05 Apr 2011 - CronJob ran 689/690 ----------- 04 Apr 2011 - CronJob ran 688/689 ----------- 03 Apr 2011 - CronJob ran 687/688 ----------- 02 Apr 2011 - CronJob ran 686/687 ----------- 01 Apr 2011 - CronJob ran 685/686 ----------- 31 Mar 2011 - CronJob ran 684/685 ----------- 30 Mar 2011 - CronJob ran 683/684 ----------- 30 Mar 2011 - CronJob ran 682/683 ----------- 30 Mar 2011 - CronJob ran 683/684 ----------- 30 Mar 2011 - CronJob ran 683/684 ----------- 29 Mar 2011 - CronJob ran 682/683 ----------- 28 Mar 2011 - CronJob ran 681/682 ----------- 27 Mar 2011 - CronJob ran 680/681 ----------- 26 Mar 2011 - CronJob ran 679/680 ----------- 25 Mar 2011 - CronJob ran 678/679 ----------- 24 Mar 2011 - CronJob ran 677/678 ----------- 23 Mar 2011 - CronJob ran 676/677 ----------- 22 Mar 2011 - CronJob ran 675/676 ----------- 21 Mar 2011 - CronJob ran 674/675 ----------- 20 Mar 2011 - CronJob ran 673/674 ----------- 19 Mar 2011 - CronJob ran 672/673 ----------- 18 Mar 2011 - CronJob ran 671/672 ----------- 17 Mar 2011 - CronJob ran 670/671 ----------- 16 Mar 2011 - CronJob ran 669/670 ----------- 15 Mar 2011 - CronJob ran 668/669 ----------- 14 Mar 2011 - CronJob ran 667/668 ----------- 13 Mar 2011 - CronJob ran 666/667 ----------- 12 Mar 2011 - CronJob ran 665/666 ----------- 11 Mar 2011 - CronJob ran 664/665 ----------- 10 Mar 2011 - CronJob ran 663/664 ----------- 09 Mar 2011 - CronJob ran 662/663 ----------- 08 Mar 2011 - CronJob ran 661/662 ----------- 07 Mar 2011 - CronJob ran 660/661 ----------- 06 Mar 2011 - CronJob ran 659/660 ----------- 05 Mar 2011 - CronJob ran 658/659 ----------- 05 Mar 2011 - CronJob ran 657/658 ----------- 05 Mar 2011 - CronJob ran 656/657 ----------- 05 Mar 2011 - CronJob ran 658/659 ----------- 04 Mar 2011 - CronJob ran 657/658 ----------- 03 Mar 2011 - CronJob ran 656/657 ----------- 02 Mar 2011 - CronJob ran 655/656 ----------- 01 Mar 2011 - CronJob ran 654/655 ----------- 28 Feb 2011 - CronJob ran 653/654 ----------- 27 Feb 2011 - CronJob ran 652/653 ----------- 26 Feb 2011 - CronJob ran 651/652 ----------- 25 Feb 2011 - CronJob ran 650/651 ----------- 24 Feb 2011 - CronJob ran 649/650 ----------- 23 Feb 2011 - CronJob ran 648/649 ----------- 22 Feb 2011 - CronJob ran 647/648 ----------- 21 Feb 2011 - CronJob ran 646/647 ----------- 20 Feb 2011 - CronJob ran 645/646 ----------- 19 Feb 2011 - CronJob ran 644/645 ----------- 18 Feb 2011 - CronJob ran 643/644 ----------- 17 Feb 2011 - CronJob ran 642/643 ----------- 16 Feb 2011 - CronJob ran 641/642 ----------- 15 Feb 2011 - CronJob ran 640/641 ----------- 14 Feb 2011 - CronJob ran 639/640 ----------- 13 Feb 2011 - CronJob ran 638/639 ----------- 12 Feb 2011 - CronJob ran 637/638 ----------- 11 Feb 2011 - CronJob ran 636/637 ----------- 10 Feb 2011 - CronJob ran 637/638 ----------- 10 Feb 2011 - CronJob ran 636/637 ----------- 10 Feb 2011 - CronJob ran 635/636 ----------- 10 Feb 2011 - CronJob ran 634/635 ----------- 10 Feb 2011 - CronJob ran 633/634 ----------- 10 Feb 2011 - CronJob ran 632/633 ----------- 10 Feb 2011 - CronJob ran 631/632 ----------- 05 Feb 2011 - CronJob ran 630/631 ----------- 04 Feb 2011 - CronJob ran 629/630 ----------- 03 Feb 2011 - CronJob ran 628/629 ----------- 02 Feb 2011 - CronJob ran 627/628 ----------- 01 Feb 2011 - CronJob ran 626/627 ----------- 31 Jan 2011 - CronJob ran 625/626 ----------- 30 Jan 2011 - CronJob ran 624/625 ----------- 29 Jan 2011 - CronJob ran 623/624 ----------- 28 Jan 2011 - CronJob ran 622/623 ----------- 27 Jan 2011 - CronJob ran 621/622 ----------- 26 Jan 2011 - CronJob ran 620/621 ----------- 25 Jan 2011 - CronJob ran 619/620 ----------- 24 Jan 2011 - CronJob ran 618/619 ----------- 23 Jan 2011 - CronJob ran 617/618 ----------- 22 Jan 2011 - CronJob ran 616/617 ----------- 21 Jan 2011 - CronJob ran 615/616 ----------- 20 Jan 2011 - CronJob ran 614/615 ----------- 19 Jan 2011 - CronJob ran 613/614 ----------- 18 Jan 2011 - CronJob ran 612/613 ----------- 17 Jan 2011 - CronJob ran 611/612 ----------- 16 Jan 2011 - CronJob ran 610/611 ----------- 15 Jan 2011 - CronJob ran 609/610 ----------- 14 Jan 2011 - CronJob ran 608/609 ----------- 13 Jan 2011 - CronJob ran 607/608 ----------- 12 Jan 2011 - CronJob ran 606/607 ----------- 11 Jan 2011 - CronJob ran 605/606 ----------- 10 Jan 2011 - CronJob ran 604/605 ----------- 09 Jan 2011 - CronJob ran 603/604 ----------- 08 Jan 2011 - CronJob ran 602/603 ----------- 07 Jan 2011 - CronJob ran 601/602 ----------- 06 Jan 2011 - CronJob ran 600/601 ----------- 05 Jan 2011 - CronJob ran 599/600 ----------- 04 Jan 2011 - CronJob ran 598/599 ----------- 03 Jan 2011 - CronJob ran 597/598 ----------- 02 Jan 2011 - CronJob ran 596/597 ----------- 01 Jan 2011 - CronJob ran 595/596 ----------- 31 Dec 2010 - CronJob ran 594/595 ----------- 30 Dec 2010 - CronJob ran 593/594 ----------- 29 Dec 2010 - CronJob ran 592/593 ----------- 28 Dec 2010 - CronJob ran 591/592 ----------- 27 Dec 2010 - CronJob ran 590/591 ----------- 26 Dec 2010 - CronJob ran 589/590 ----------- 25 Dec 2010 - CronJob ran 588/589 ----------- 24 Dec 2010 - CronJob ran 587/588 ----------- 23 Dec 2010 - CronJob ran 586/587 ----------- 22 Dec 2010 - CronJob ran 585/586 ----------- 21 Dec 2010 - CronJob ran 584/585 ----------- 20 Dec 2010 - CronJob ran 583/584 ----------- 19 Dec 2010 - CronJob ran 582/583 ----------- 18 Dec 2010 - CronJob ran 581/582 ----------- 17 Dec 2010 - CronJob ran 580/581 ----------- 16 Dec 2010 - CronJob ran 579/580 ----------- 15 Dec 2010 - CronJob ran 578/579 ----------- 14 Dec 2010 - CronJob ran 577/578 ----------- 13 Dec 2010 - CronJob ran 576/577 ----------- 12 Dec 2010 - CronJob ran 575/576 ----------- 11 Dec 2010 - CronJob ran 574/575 ----------- 10 Dec 2010 - CronJob ran 573/574 ----------- 09 Dec 2010 - CronJob ran 572/573 ----------- 08 Dec 2010 - CronJob ran 571/572 ----------- 07 Dec 2010 - CronJob ran 570/571 ----------- 06 Dec 2010 - CronJob ran 569/570 ----------- 05 Dec 2010 - CronJob ran 568/569 ----------- 04 Dec 2010 - CronJob ran 567/568 ----------- 03 Dec 2010 - CronJob ran 566/567 ----------- 02 Dec 2010 - CronJob ran 565/566 ----------- 01 Dec 2010 - CronJob ran 564/565 ----------- 30 Nov 2010 - CronJob ran 563/564 ----------- 29 Nov 2010 - CronJob ran 562/563 ----------- 28 Nov 2010 - CronJob ran 561/562 ----------- 27 Nov 2010 - CronJob ran 560/561 ----------- 26 Nov 2010 - CronJob ran 559/560 ----------- 25 Nov 2010 - CronJob ran 558/559 ----------- 24 Nov 2010 - CronJob ran 557/558 ----------- 23 Nov 2010 - CronJob ran 556/557 ----------- 22 Nov 2010 - CronJob ran 555/556 ----------- 21 Nov 2010 - CronJob ran 554/555 ----------- 20 Nov 2010 - CronJob ran 553/554 ----------- 19 Nov 2010 - CronJob ran 552/553 ----------- 18 Nov 2010 - CronJob ran 551/552 ----------- 17 Nov 2010 - CronJob ran 550/551 ----------- 16 Nov 2010 - CronJob ran 549/550 ----------- 15 Nov 2010 - CronJob ran 548/549 ----------- 15 Nov 2010 - CronJob ran 547/548 ----------- 15 Nov 2010 - CronJob ran 546/547 ----------- 12 Nov 2010 - CronJob ran 545/546 ----------- 11 Nov 2010 - CronJob ran 544/545 ----------- 10 Nov 2010 - CronJob ran 543/544 ----------- 09 Nov 2010 - CronJob ran 542/543 ----------- 08 Nov 2010 - CronJob ran 541/542 ----------- 07 Nov 2010 - CronJob ran 540/541 ----------- 06 Nov 2010 - CronJob ran 539/540 ----------- 05 Nov 2010 - CronJob ran 538/539 ----------- 04 Nov 2010 - CronJob ran 537/538 ----------- 03 Nov 2010 - CronJob ran 536/537 ----------- 02 Nov 2010 - CronJob ran 535/536 ----------- 01 Nov 2010 - CronJob ran 534/535 ----------- 31 Oct 2010 - CronJob ran 533/534 ----------- 30 Oct 2010 - CronJob ran 532/533 ----------- 29 Oct 2010 - CronJob ran 531/532 ----------- 28 Oct 2010 - CronJob ran 530/531 ----------- 27 Oct 2010 - CronJob ran 529/530 ----------- 26 Oct 2010 - CronJob ran 528/529 ----------- 25 Oct 2010 - CronJob ran 527/528 ----------- 24 Oct 2010 - CronJob ran 526/527 ----------- 23 Oct 2010 - CronJob ran 525/526 ----------- 22 Oct 2010 - CronJob ran 524/525 ----------- 21 Oct 2010 - CronJob ran 523/524 ----------- 20 Oct 2010 - CronJob ran 522/523 ----------- 19 Oct 2010 - CronJob ran 521/522 ----------- 18 Oct 2010 - CronJob ran 520/521 ----------- 17 Oct 2010 - CronJob ran 519/520 ----------- 16 Oct 2010 - CronJob ran 518/519 ----------- 15 Oct 2010 - CronJob ran 517/518 ----------- 14 Oct 2010 - CronJob ran 516/517 ----------- 13 Oct 2010 - CronJob ran 515/516 ----------- 12 Oct 2010 - CronJob ran 514/515 ----------- 11 Oct 2010 - CronJob ran 513/514 ----------- 11 Oct 2010 - CronJob ran 512/513 ----------- 10 Oct 2010 - CronJob ran 511/512 ----------- 08 Oct 2010 - CronJob ran 510/511 ----------- 07 Oct 2010 - CronJob ran 509/510 ----------- 06 Oct 2010 - CronJob ran 508/509 ----------- 05 Oct 2010 - CronJob ran 507/508 ----------- 04 Oct 2010 - CronJob ran 506/507 ----------- 03 Oct 2010 - CronJob ran 505/506 ----------- 02 Oct 2010 - CronJob ran 504/505 ----------- 01 Oct 2010 - CronJob ran 503/504 ----------- 30 Sep 2010 - CronJob ran 502/503 ----------- 29 Sep 2010 - CronJob ran 501/502 ----------- 28 Sep 2010 - CronJob ran 500/501 ----------- 27 Sep 2010 - Sunil ran 499/500 - all OK ----------- 27 Sep 2010 - CronJob ran 499/500 ----------- 26 Sep 2010 - CronJob ran 498/499 ----------- 25 Sep 2010 - CronJob ran 497/498 ----------- 24 Sep 2010 - CronJob ran 496/497 ----------- 23 Sep 2010 - CronJob ran 495/496 ----------- 22 Sep 2010 - CronJob ran 494/495 ----------- 21 Sep 2010 - CronJob ran 493/494 ----------- 20 Sep 2010 - CronJob ran 492/493 ----------- 19 Sep 2010 - CronJob ran 491/492 ----------- 18 Sep 2010 - CronJob ran 490/491 ----------- 17 Sep 2010 - CronJob ran 489/490 ----------- 16 Sep 2010 - CronJob ran 488/489 ----------- 15 Sep 2010 - CronJob ran 487/488 ----------- 14 Sep 2010 - CronJob ran 486/487 ----------- 13 Sep 2010 - CronJob ran 485/486 ----------- 12 Sep 2010 - CronJob ran 484/485 ----------- 11 Sep 2010 - CronJob ran 483/484 ----------- 10 Sep 2010 - CronJob ran 482/483 ----------- 09 Sep 2010 - CronJob ran 481/482 ----------- 08 Sep 2010 - CronJob ran 480/481 ----------- 07 Sep 2010 - CronJob ran 479/480 ----------- 06 Sep 2010 - CronJob ran 478/479 ----------- 05 Sep 2010 - CronJob ran 477/478 ----------- 04 Sep 2010 - CronJob ran 476/477 ----------- 03 Sep 2010 - CronJob ran 475/476 ----------- 02 Sep 2010 - CronJob ran 474/475 ----------- 01 Sep 2010 - CronJob ran 473/474 ----------- 31 Aug 2010 - CronJob ran 472/473 ----------- 30 Aug 2010 - CronJob ran 471/472 ----------- 29 Aug 2010 - CronJob ran 470/471 ----------- 28 Aug 2010 - CronJob ran 469/470 ----------- 27 Aug 2010 - CronJob ran 468/469 ----------- 26 Aug 2010 - CronJob ran 467/468 ----------- 25 Aug 2010 - CronJob ran 466/467 ----------- 24 Aug 2010 - CronJob ran 465/466 ----------- 23 Aug 2010 - CronJob ran 464/465 ----------- 22 Aug 2010 - CronJob ran 463/464 ----------- 21 Aug 2010 - CronJob ran 462/463 ----------- 20 Aug 2010 - CronJob ran 461/462 ----------- 19 Aug 2010 - CronJob ran 460/461 ----------- 18 Aug 2010 - CronJob ran 459/460 ----------- 17 Aug 2010 - CronJob ran 458/459 ----------- 16 Aug 2010 - CronJob ran 457/458 ----------- 15 Aug 2010 - CronJob ran 456/457 ----------- 14 Aug 2010 - CronJob ran 455/456 ----------- 13 Aug 2010 - CronJob ran 454/455 ----------- 12 Aug 2010 - CronJob ran 453/454 ----------- 11 Aug 2010 - CronJob ran 452/453 ----------- 10 Aug 2010 - CronJob ran 451/452 ----------- 09 Aug 2010 - CronJob ran 450/451 ----------- 08 Aug 2010 - CronJob ran 449/450 ----------- 07 Aug 2010 - CronJob ran 448/449 ----------- 06 Aug 2010 - CronJob ran 447/448 ----------- 5 Aug 2010 - ok ran 446/447 ----------- 4 Aug 2010 - ok ran 445/446 ----------- 3 Aug 2010 ran 444/445 - ok ----------- 2 Aug 2010 ran 443/444 - ok ----------- 1 Aug 2010 ran 443/443 - ok ----------- 31 Jul 2010 ran 441/442 - ok ----------- 30 Jul 2010 ran 440/441 - ok ----------- 29 Jul 2010 ran 439/440 - ok ----------- 28 Jul 2010 ran 438/439 - ok ----------- 27 Jul 2010 ran 437/438 - ok ----------- 26 Jul 2010 ran 436/437 ----------- 25 Jul 2010 ran 435/436 - ok ----------- 24 Jul 2010 ran 434/435 - ok ----------- 23 Jul 2010 ran 433/434 - ok ---------- 22 Jul 2010 ran 432/433 OD433 - all packets here, MemCheck problem (expected and to be fixed) ----------- 21 Jul 2010 ran 431/432 all packets here, MemCheck problem (expected and to be fixed) ----------- 20 Jul 2010 Sunil ran 428/429/430/431 430 - 600 CHK packets missing ----------- 19 Jul 2010 ran 428/429/430 430 - ok 429 - missing 3300 CHK 428 - missing 300 CHK ---------- 18 Jul 2010 ran 428/429 429 - ok 428 missing a few CHK ---------- 17 Jul 2010 ran 427/428 - ok ----------- 16 Jul 2010 Sunil ran 426/427 - OK ----------- 15 Jul 2010 Sunil ran 424/425/426 - OK received missing CHK packets from OD 424 ----------- 14 Jul 2010 Sunil ran 424/425 - ok but 300 missing CHK packets from near the end of OD 424 (after transition from PHOT_STBY to REDY) ----------- 13 Jul 2010 Sunil ran 423/424 - ok ----------- 12 Jul 2010 Sunil ran 422/423 - ok ----------- 11 Jul 2010 ran 421/422 - ok ------------ 10 Jul 2010 ran 420/421 - ok 419 missinf CHK arived ---------- 09 Jul 2010 ran 419/420 419 - ~6000 CHK missing 420 - ok ---------- 08 Jul 2010 ran 418/419 - ok OD413 - still missing 900 CHK OD416 - still appear to be missing all science data OD417 - still missing ~2000 SuSD ---------- 07 Jul 2010 ran 411->418 OD413 - still missing 900 CHK OD414 - ok OD415 - arrived ok OD416 - all Science Data missing OD417 - [OBS DAY], Seems all ok , Except missing SuSD gaps. OD418 - Seems ok ---------- 06 Jul 2010 ran 411 -> 417 - problems! ---------- 05 Jul 2010 ran 413/414/415/416 416 - all packets missing 415 - most packets missing 413 - still missing CHK ---------- 04 Jul 2010 ran 413/414/415 415 - most packets missing 413 - still missing CHK --------- 03 Jul 2010 ran 413/414 414 ok 413 still missing CHK ---------- 02 Jul 2010 ran 412/413 - missing CHK on OD413 ----------- 01 Jul 2010 Sunil Reran OD411 after MOC forwarded the consolidated TM to the HSC All missing PDT received OK ------------ 01 Jul 2010 ran 410/411/412 - missing PDT on OD411 ------------ 29 Jun 2010 ran 409/410 - ok ------------ 28 Jun 2010 ran 408/409 - ok ------------ 27 Jun 2010 ran 407/408 - ok ------------ 26 Jun 2010 ran 406/407 - ok ---------- 25 Jun 2010 ran 405/406 - ok ---------- 24 Jun 2010 ran 404/405 - ok ---------- 23 Jun 2010 ran 403/404 - ok ---------- 22 Jun 2010 ran 402/403 - ok ---------- 21 Jun 2010 ran 401/402 - ok ---------- 20 Jun 2010 ran 399/400/401 - ok ---------- 18 Jun 2010 ran 398/399 - ok ---------- 17 Jun 2010 ran 397/398 - ok ---------- 16 Jun 2010 - ok ran 396/397 ---------- 15 Jun 2010 ran 395/396 - ok ---------- 14 Jun 2010 ran 394/395 - ok ---------- 13 Jun 2010 ran 393/394 - ok ---------- 12 Jun 2010 ran 392/393 - ok ---------- 11 Jun 2010 ran 391/392 - ok ---------- 10 Jun 2010 ran 390/391 - ok ---------- 09 Jun 2010 ran 389/390 - ok ----------- 08 Jun 2010 ran 388-389 - ok -------------- 07 Jun 2010 ran 387-388 - all ok --------- 06 Jun 2010 ran 386 - 387 - all ok --------- 05 Jun 10 ran 385 - 386 - all ok --------- 04 Jun 10 ran 384 - 385 - all ok ---------- 03 Jun 10 ran 383 - 384 - all ok ----------- 02 Jun 2010 ran 382- 383 - all ok ----------- 01 Jun 2010 Andreas reran 381 ran 382 - ok ------------ 31 May 2010 Andreas reran 380 ran 381 - ok -------------- 30 May 2010 Andreas reran 379 ran 380 - ok --------- 29 May 2010 Andreas reran 378 ran 379 - ok ---------- 28 May 2010 Andreas reran 377 ran 378 - ok ---------- 27 May 2010 Andreas reran 376 ran 377 - ok --------- 26 May 2010 Andreas reran 375 ran 376 ok ----------- 25 May 2010 Andreas reran 374 ran 375 - ok ---------- 24 May 2010 Andreas reran 373 ran 374 - ok ---------- 23 May 2010 Andreas reran 372 ran 373 - ok ------------ 22 May 2010 Andreas reran 371 ran 372 - ok ----------- 21 May 2010 Andreas reran 379 ran 371 - ok ----------- 20 May 2010 Andreas reran 369 ran 370 - ok ----------- 19 May 2010 Andreas reran 368 ran 369 ---------- 18 May 2010 Andreas reran 367 ran 368 ---------- 17 May 2010 Andreas reran 366 ran 367 - ok ----------- 16 May 2010 Andreas reran 365 ran 366 - ok ----------- 15 May 2010 Andreas reran 364 ran 365 - ok ----------- 14 May 2010 Andreas reran 363 ran 364 - ok ------------ 13 May 2010 Andreas reran 362 ran 363 - ok ------------ 12 May 2010 Andreas reran 361 ran 362 - ok ------------ 11 May 2010 Andreas reran 360 ran 361 - ok ------------ 10 May 2010 Andreas reran 359 ran 360 - ok ------------- 9 May 2010 Andreas reran 358 ran 359 - ok -------------- 8 May 2010 Sunil reran 356 ran 357 - ok ran 358 - ok --------------- 6 May 2010 Andreas reran 355 ran 356 - ok --------------- 5 May 2010 Andreas reran 354 ran 355 - ok ---------------- 4 May 2010 Andreas ran 353 reran 354 - ok --------------- 3 May 2010 Andreas ran 352 reran 353 - ok --------------- 2 May 2010 Andreas reran 351 ran 352 - ok ---------------- 1 May 2010 Andreas reran 350 ran 351 - all ok ---------------- 30 Apr Andreas reran 349 ran 350 - all ok ---------------- 29 Apr 2010 Andreas reran 348 ran 349 - all ok ----------------- 28 April 2010 Ed - updated HealthMonitor.py to CVS 1.29 - now SMEC check and Mem check packets will only appear on the web page if those packets were produced. re-ran for ODs345-348 --------------------------- 27 Apr 2010 Andreas reran 344 - mising chk packets recovered reran 346 ran 347 - all ok ---------------- 26 Apr 2010 Andreas reran 345 ran 346 - all ok --------------- 25 Apr 2010 Andreas reran 343 reran 344 - 2100 CHK packets missing ran 345 - data arrived ok - some programMemCheck flags --------------- 24 Apr 2010 Andreas reran 343 ran 344 - some CHK packets missing, will reran later --------------- 23 Apr 2010 Andreas reran 342 ran 343 - all ok ---------------- 22 Apr 2010 Andreas reran 341 ran 342 - all ok, some expected DRCY events ---------------- 21 Apr 2010 Andreas reran 340 ran 341 - all ok, some expected DRCU events ---------------- 20 Apr 2010 Andreas reran 339 ran 340 - all ok ---------------- 19 Apr 2010 Andreas reran 334 ran 335 - all ok ran 336 - all ok ran 337 - all ok ran 338 - all ok ran 339 - all ok ran 340 - no data --------------- 15 April 2010 Andreas reran 334 ran 335 ------------ 14 April 2010: Ed updated HealthMonitor script to v1.27 - SMEC loop and mem check packets now on web page. ------------- 13 April 2010 Sunil reran 333 ran 334 - all OK ------------- 12 April 2010 Ed reran 329 ran 330-332 Looks like propagation not complete yet for OD332 - there are only NHK and no CHK yet. Otherwise all OK. Sunil reran OD331 ran 332 - all OK -------------- 9 April 2010 Sunil reran 328 ran 329 - all OK ------------- 8 April 2010 Sunil reran 327 ran 328 - all OK ------------- 7 April 2010 Sunil reran 326 ran 327 - all OK (SCAL2 was used and seems to have worked OK) ------------- 6 April 2010 Ed reran 325 ran 326 - 344 transparent packets. Some MCU frame ID events. All occur during BSM tuning observation. Sunil says should be OK. obsLog - problem initially because user.props set to Network connection, but OK after changing to LocalConnection. ------------- 5 Apr 2010 Andreas reran 324 ran 325 - all ok --------------- 4 Apr 2010 Sunil reran 323 ran 324 - all ok --------------- 3 Apr 2010 Andreas reran 322 ran 323 - all ok --------------- 2 Apr 2010 Andreas reran 321 ran 322 - all ok --------------- 1 Apr 2010 Andreas reran 320 ran 321 - all ok ------------- 31 Mar 2010 Andreas reran 319 ran 320 - all ok ------------- 30 Mar 2010 Andreas reran 318 ran 319 - all ok -------------- 29 Mar 2010 Andreas reran 317 ran 318 - all ok -------------- 28 Mar 2010 Andreas reran 316 ran 317 -------------- 27 Mar 2010 Andreas reran 315 ran 316 - Houskeeping ok - Science data missing (correction: no Science data data were expected) ------------- 26 Mar 2010 Andreas reran 314 ran 315 - all ok ------------- 25 Mar 2010 Andreas reran 313 ran 314 - all ok ------------- 24 Mar 2010 Andreas reran 312 ran 313 - expected DCU errors - all ok -------------- 23 Mar 2010 Andreas reran 311 ran 312 - all ok --------------- 22 Mar 2010 Andreas reran 301 ran 311 - all ok ----------- 21 Mar 2010 Andreas reran 308-309 ran 310 - all ok ------------ 19 Mar 2010 Andreas reran 307 ran 308 - all ok ------------- 18 Mar 2010 Andreas reran 306 ran 307 - all ok -------------- 17 Mar 2010 Andreas reran 305 ran 306 - all ok --------------- 16 Mar 2010 Andreas reran 301,302,303,304 - Data are in database now. ran 305 - all ok --------------- 14 Mar 2010 reran 302 - STILL NO CRITICAL PACKETS ran 303- NO DATA RECEIVED ---------------- 13 Mar 2010 reran 301 ran 302 - NO CRITICAL PACKETS RECEIVED --------------- 12 Mar 2010 Andreas reran 300 ran 301 - loads of expected DCU events - all ok --------------- 11 Mar 2010 Andreas reran 299 ran 300 - all ok -------------- 10 Mar 2010 Andreas reran 298 ran 299 - all ok -------------- 09 Mar 2010 Andreas reran 297 ran 298 - all ok --------------- 08 Mar 2010 Andreas reran 296 ran 297 - all ok ---------------- 07 Mar 2010 Andreas reran 295 ran 296 - all ok ----------------- 06 Mar 2010 Andreas reran 294 ran 295 - all ok ------------------ 05 Mar 2010 Andreas reran 293 ran 294 - all ok ------------------ 04 Mar 2010 Andreas reran 292 ran 293 - all ok ------------------- 03 Mar 2010 Andreas reran 291 ran 292 - all ok -------------------- 02 Mar 2010 Markos reran 290 ran 291- -no events, no missing packets --------------------- 01 Mar 2010 Markos reran 289 ran 290- -85 events --------------------- 28 Feb 2010 Andreas reran 288 ran 289- -all ok --------------------- 27 Feb 2010 Andreas rernan 287 ran 288 - all ok ------------------------- 26 Feb 2010 Andreas reran 286 ran 287 - all ok ------------------------------- 25 Feb 2010 Sunil reran 285 ran 286 - No problems ---------------------------------- 24 Feb 2010 Sunil reran 284 ran 285 - No problems Yesterday's ObsLog error was caused by a missing log file for OD0216. Quite why this file was missing is a mystery. Ed reran ObsLog for OD0216 and then for OD0284 without any problems. ---------------------------------- 23 Feb 2010 Sunil reran 283 ran 284 - No problems ObsLog generated an error: s_ObsLog_DailyLog_0269', 'SpireOps_ObsLog_DailyLog_0270', 'SpireOps_ObsLog_DailyLog_0271', 'SpireOps_ObsLog_DailyLog_0272', 'SpireOps_ObsLog_DailyLog_0273', 'SpireOps_ObsLog_DailyLog_0274', 'SpireOps_ObsLog_DailyLog_0275', 'SpireOps_ObsLog_DailyLog_0276', 'SpireOps_ObsLog_DailyLog_0277', 'SpireOps_ObsLog_DailyLog_0278', 'SpireOps_ObsLog_DailyLog_0279', 'SpireOps_ObsLog_DailyLog_0280', 'SpireOps_ObsLog_DailyLog_0281', 'SpireOps_ObsLog_DailyLog_0282', 'SpireOps_ObsLog_DailyLog_0283', 'SpireOps_ObsLog_DailyLog_0284', 'SpireOps_ObsLog_DailyLog_0287', 'SpireOps_ObsLog_DailyLog_0288'] Exception in thread "Thread-2" Traceback (innermost last): File "/misc/winchester1/ops/operations/scripts/./ObservationLog.py", line 295, in run File "/misc/winchester1/ops/operations/scripts/./ObservationLog.py", line 376, in combineLogs File "/misc/winchester1/ops/operations/scripts/./ObservationLog.py", line 475, in addLog UnboundLocalError: local: 'chosenVersion' TA ran OK but noticed that there are regular warnings issued which have probably been there all the time but highlighted here in case they cause problems later) 23-Feb-10 10:13:10.227 INFO LocalConnection: Making a local connection to ops_1@chesterfield 23-Feb-10 10:13:10.294 INFO QueryLocator: Query completed, size unknown, execution time 56ms 23-Feb-10 10:13:17.610 INFO PoolCreatorFactory: Creating pool SpireOps_TrendAnalysis of type lstore from defaults 23-Feb-10 10:13:43.611 WARNING FitsProductPool: Conrruptted URN founded: urn:SpireOps_TrendAnalysis:herschel.ia.dataset.Product:10, suggest to run rebuildIndex(). 23-Feb-10 10:13:43.687 WARNING FitsProductPool: Conrruptted URN founded: urn:SpireOps_TrendAnalysis:herschel.ia.dataset.Product:18, suggest to run rebuildIndex(). 23-Feb-10 10:13:44.759 WARNING FitsProductPool: Conrruptted URN founded: urn:SpireOps_TrendAnalysis:herschel.ia.dataset.Product:42, suggest to run rebuildIndex(). 23-Feb-10 10:13:45.146 WARNING FitsProductPool: Conrruptted URN founded: urn:SpireOps_TrendAnalysis:herschel.ia.dataset.Product:50, suggest to run rebuildIndex(). ---------------------------------- 22 Feb 2010 Ed reran 282 ran 283 - No problems ---------------------------------- 21 Feb 2010 Andreas ---------------------------------- 22 Feb 2010 Ed reran 282 ran 283 - No problems ---------------------------------- 21 Feb 2010 Andreas reran 281 ran 282 - No problems ---------------------------------- 20 Feb 2010 Andreas reran 280 ran 281 - no problems ----------------------------------- 19 Feb 2010 Andreas reran 279 ran 280 - no problems --------------------------------------------------------------- 18 Feb 2010 Andreas reran 278 ran 279: no problems ------------------------------------------------------------------- 17 Feb 2010 Andreas reran 277 ran 278: no problems ------------------------------------------------------------------- 16 Feb 2010 Ed re-ran OD276 ran OD277: no problems -------------------------------------------------------------------- 15 Feb 2010 Ed re-ran OD272 ran OD273-276: OD274: expected no response and N->W errors OD275: data seem to stop at some point - no critical packets at all OD276: no data (problem at HSC side) reran, OD275 and OD276 after data had been propagated. No problems. -------------------------------------------------------------------- 11 Feb 2010 Markos re-ran OD271 ran OD272, no events, no missing packets -------------------------------------------------------------------- 10 Feb 2010 Markos re-ran OD270 (2 events) ran OD271, no events, no missing packets -------------------------------------------------------------------- 9 Feb 2010 Ed re-ran OD269, ran OD270. One expected N->W event. -------------------------------------------------------------------- 8 Feb 2010 Sunil Reran OD267 - no missing packets Ran ODs 268-269 - no missing packets 2 "expected" L1 temperature warning events in OD268 Lots of "expected" No DCU/MCU/SCU Response Errors during POF2 observations in OD269 (while chopping) -------------------------- 5 Feb 2010 Andreas reran 265 ran 266 - No missing packets - Some expected events -------------------------- 4 Feb 2010 Andreas reran 264 ran 265 N missing packets ** DRCU EVENT:ERR TC_SEQ_ERROR ** 4 Command Failures -------------------------- 3 Feb 2010 Ed reran 263 ran 264: no problems. ran future obs log for 268,269,270,271, 273 --------------------------- 2 Feb 2010 Ed reran 262 ran 263: no problems. --------------------------- 1 Feb 2010 Ed reran 261 ran 262: no problems. --------------------------- 31 Jan 2010 Andreas Reran 260 Ran 261 - No missing packets - One expected event --------------------------- 30 Jan 2010 Andreas Reran 259 Ran 260 - No Missing Packets - Some expected DRCU Also on 260 - LS Interface Overflow event which is subsequently cleared. ---------------------------- 29 Jan 2010 Andreas Reran 258 Run 259 - No Missing packets - Some expected DRCU events ------------------------------ 28 Jan 2010 Andreas Rerun 257 Run 258 - No missing packets - No events ------------------------------- 27 Jan 2010 Ken Reran 256- No missing packets - No events. Ran 257 - No missing packets - No events. ------------------------------- 26 Jan 2010 Markos Reran 255- No missing packets - No events. Ran 256 - No missing packets - No events. ------------------------------- 25 Jan 2010 Markos Reran 254- No missing packets - No events. Ran 255 - No missing packets - No events. ------------------------------- 24 Jan 2010 Andreas Rerad 253 Ran 254 - No missing packets - No events. ------------------------------- 23 Jan 2010 Andreas I presume someone run 252. (Monitoring exists on web page) Reran 252 Ran 253 - No missing packets - expected DRCU events -------------------------------- 21 Jan 2010 Sunil Reran OD250 Ran OD251 - No missing packets. - No temperature warning events from the monitoring system as L1 temperature remained above 4K. --------------------------------- 20 Jan 2010 Sunil Reran OD249 Ran OD250 - No missing packets. Expected DRCU response error/clear events. - No temperature warning events from the monitoring system as L1 temperature remained above 4K. --------------------------------- 19 Jan 2010 Ken Reran OD248 Ran OD249 - No missing packets. Expected a8e8 DRCU events. --------------------------------- 18 Jan 2010 Ken Reran OD247 Ran OD248 - No missing packets. Expected a8e8 DRCU events. --------------------------------- 17 Jan 2010 Andreas Reran OD246 - ran OD247. No missing packets. Expected a8e8 DRCU events. --------------------------------- 16 Jan 2010 Andreas Reran OD245 - ran OD246. No missing packets. Expected a8e8 DRCU events. --------------------------------- 15 Jan 2010 Ed Reran OD244 and ran OD245. No missing packets. Expected a8e8 DRCU events. --------------------------------- 14 Jan 2010 Ed Reran OD243 and ran OD244. No missing packets. No Events. --------------------------------- 13 Jan 2010 Ed Reran OD242 and ran OD243. No missing packets. Expected a8e8 DRCU events. --------------------------------- 12 Jan 2010 Andreas reran OD241 ran OD242 - No missing packets - Expected DRCU events --------------------------------- 11 Jan 2010 Andreas ran OD239 - 240 - 241 no missing packets - expected DRCU events --------------------------------- 8 Jan 2010 Sunil reran OD236 ran ODs 237 and 238 - no missing packets - expected DRCU events ------------------------------------ 6 Jan 2010 Ken reran OD235 ran OD 236 - no missing packets - expected DRCU events ------------------------------------ 5 Jan 2010 Ken reran OD234 ran OD235 - no missing packets - expected DRCU events ------------------------------------ 4 Jan 2010 Ken reran OD233 ran OD234 - no missing packets - expected DRCU events ------------------------------------ 3 Jan 2010 Andreas reran 232 ran 233 - no missing packets - some expected DRCU events ------------------------------------ 2 Jan 2010 Andreas reran OD231 ran OD232 - no missing packets - loads of "no SCU/DCU/MCU response" that are latered cleared. ------------------------------------- 1 Jan 2010 Andreas Happy New year! reran OD230 ran OD231 - no missing packets - some expected DRCU events -------------------------------------- 31 Dec 2009 Sunil re-ran OD229 ran OD230 - no missing packets ---------------------------------------- 30 Dec 2009 Andreas reran 228 ran 229 - no missing packets ---------------------------------------- 29 Dec 2009 Andreas reran 227 - no missing packets. ran 228 - no problems, some expected events ------------------------------------------ 28 Dec 2009 Ed reran 226, ran 227 some missing NHK packets. some events on OD 226 but all expected: due to temperature limits (e.g. ERR N->W Cmd: a8e30000) or no response errors. -------------------------------------------- 27 Dec 2009 Andreas re-ran OD225 ran OD226 - no missing packets ------------------------------------------- 26 Dec 2009 Andreas ran 225 - no missing packets --------------------------------------------- 25 Dec 2009 Andreas re-ran OD223 ran OD224 - no missing packets ---------------------------------------------- 24 Dec 2009 Sunil re-ran 0D222 - No problems ran OD223 - no missing packets ------------------------------------------------ 23 Dec 2009 Andreas ran 0D222 - No problems re-ran OD221 - no missing packets ------------------------------------------------ 22 Dec 2009 Ken re-ran OD220 - no missing packets ran OD221 - no problems ------------------------------------------------- 21 Dec 2009 Ken re-ran OD217 ran ODs 218, 219, 220 - 39 missing NHK packets in OD220 ------------------------------------------------- 17 Dec 2009 Sunil Andreas had already run OD216 - 472 missing CHK packets re-ran OD215 re-ran OD216 - missing CHK packets recovered ------------------------------------------------- 16 Dec 2009 Ken re-ran OD214 ran OD215 - no problems ------------------------------------------------- 15 Dec 2009 Ken re-ran OD213 ran OD214 - no probs ------------------------------------------------- 14 Dec 2009 Ken re-ran OD210 ran Ods 211, 212, 213 - no probs ------------------------------------------------- 11 Dec 2009 Ed re-ran OD209 Health Monitoring ran OD210 - missing NHK packets from yesterday have been recovered. ------------------------------------------------- 10 Dec 2009 Ed re-ran OD208 Health Monitoring ran OD209 - 71 missing NHK packets. HSC don't see any missing packets. ------------------------------------------------- 9 Dec 2009 Ken re-ran OD207 Health Monitoring ran OD208 - no problems ------------------------------------------------- 8 Dec 2009 Ken re-ran OD206 ran OD207 - no probs Unable to generate any level 1 TA products for Ods 187, 188 ------------------------------------------------- 7 Dec 2009 Ken re-ran OD203 Ran ODs 204,205, 206 - no probs Ran Lvl1 TA for 199 Ran TA for 192, 197, 198 ------------------------------------------------- 4 Dec 2009 Ed Re-ran 202, ran 203 ------------------------------------------------- 3 Dec 2009 Ed Re-ran 201, ran 202. no probs. ------------------------------------------------- 2 Dec 2009 Ed Re-ran 200, ran 201. no probs. ------------------------------------------------- 1 Dec 2009 Ed re-ran OD199, ran OD200 several events at end but these are known ones (related to temperature limits) ------------------------------------------------- 30 Nov 2009 Markos ran 196 first, 1192 events 197, 36 events 198, fine 199, 28 events 200, not propagated yet ------------------------------------------------- 27 Nov 2009 Sunil Re-ran OD195 and then OD196 - no missing packets. 128 monitoring events are as expected. 420 MCU BSM Frame Checksum Errors and 62 MCU BSM Frame ID Errors during the SpireEngBsmTuner test. 1475 transparent packets. Similar errors were also observed during the tests on FS but far fewer in number, viz. 28 MCU BSM Frame Checksum Errors, 4 MCU BSM Frame ID Errors and 105 transparent packets To be investigated further but not considered as a problem in analysing the test data. ------------------------------------------------- 26 Nov 2009 Sunil Re-ran OD194 and then OD195 - no missing packets. 8 monitoring events are as expected. ------------------------------------------------- 25 Nov 2009 Ed Tried to re-run past days to get rid of missing packets, and ended up going right back to OD189. So re-ran OD189-194. Now no missing packets. ------------------------------------------------- 24 Nov 2009 Run OD-190 to 193 30 critical packets missing OD193 ------------------------------------------------- 20 Nov 2009 23:30 Sunil Re-ran OD188 health monitoring. Ran monitoring for OD189 despite not having received a "Propagation Completed" message. Completed successfully. No missing packets. All 27 monitoring events are understood (SCR-2187 raised). SCAL VM science packets received OK Loop remained closed for all SMEC scan observation Spectrometer peak-up observation completed OK - mode mis-match between the MIB and CUS script -------------------------------------------------- 20 Nov 2009 Ed Problems with propagation so data from OD189 not received in morning. Ran monitoring at end of day, and still no critical packets received. Needs to be run again on Monday. -------------------------------------------------- 19 Nov 2009 Markos re-ran OD187. still 1 command failure, nearly an hour of missing ran OD188 No problems. -------------------------------------------------- 18 Nov 2009 Ed re-ran OD186. Ran OD187 1 command failure, nearly an hour of missing housekeeping.. -------------------------------------------------- 17 Nov 2009 Markos Re-ran OD185 and then ran OD 186. No problems. -------------------------------------------------- 16 Nov 2009 Ed Re-ran OD182 and then ran OD 183-185. No problems. -------------------------------------------------- 13 Nov 2009 Sunil Re-ran OD181 - OK Ran OD182 - OK -------------------------------------------------- 12 Nov 2009 Sunil Re-ran OD180 - OK Ran OD181 - OK Monitoring tasks took a long time to complete. -------------------------------------------------- 11 Nov 2009 Sunil Re-ran OD179 - OK Ran OD180 - OK -------------------------------------------------- 10m Nov 2009 Ken Re-ran OD178 - OK Ran OD179 - OK Launched ok, pid = 22248 TM gap now filled -------------------------------------------------- 10m Nov 2009 Ken Re-ran OD178 - OK Ran OD179 - OK Re-ran OD176 - subsystem TM gap now filled -------------------------------------------------- 09 Nov 2009 Ken re-ran OD175 - OK Ran ODs 176, 177, 178 - Subssytem TM gap on OD176, otherwise no unexpected events -------------------------------------------------- 06 Nov 09 Ed re-ran OD173, 174. Ran OD175. Missing critical packets recovered. -------------------------------------------------- 05 Nov 2009 Andreas ran OD173 - OK ran OD174 - 30 critical missing -------------------------------------------------- 03 Nov 2009 Markos ran OD172 900 critical packets missing reran 172 missing packets dissapeared --------------------------------------------------- 02 Nov 2009 Ken reran OD168 ran OD 169, 170, 171 - no problems --------------------------------------------------- 30 Oct 2009 Ed/Sunil reran OD167. ran OD 168 - no problems --------------------------------------------------- 29 Oct 2009 Ed reran OD166. ran OD 167 - no problems - there is an LS Interface Overflow event during REDY to phot STBY, but this was cleared, and Sunil says it is nothing to worry about. --------------------------------------------------- 28 Oct 09 Ed reran OD165 ran OD166 - no problems ---------------------------------------------------- 27 Oct 2009 Ken re-ran OD164 ran OD 165 - no problems ---------------------------------------------------- 26 Oct 2009 Ken re-ran ODs 160,161 to clear HK gaps ran ODs 162, 163, 164 - no problems ---------------------------------------------------- Ed Re-ran obsLog for all ODs with new CVS version 1.38 (using RunAllODs.py) ---------------------------------------------------- 21 Oct 2009 Ken re-ran OD 158 ran OD 159 - no problems ---------------------------------------------------- 20 Oct 2009 Ken re-ran ODs 155, 156, 157 ran OD 158 no missing packets or events ---------------------------------------------------- 19 October 2009 Ed re-ran OD 154. ran ODs 155 - 157 OD 155: TC execution failures ---------------------------------------------------- 16 Oct 2009 Sunil Ran OD154 - 849 missing CHK packets from the end of OD153 Re-ran OD154 - missing CHK packets recovered ---------------------------------------------------- 15 Oct 2009 Sunil Ran OD153 - missing CHK packets from the end of OD152 Re-ran OD152 - missing CHK packets recovered ---------------------------------------------------- 14 Oct 2009 Sunil Ran OD152. No problems ---------------------------------------------------- 13 Oct 2009 Markos Ran OD151. 4 TC execution failures, 2 events, 5405 CHK packets missing. According to Tanya MOC is still updating the onboard software. ---------------------------------------------------- 12 Oct 2009 Ed re-ran OD149. Missing CHK packets recovered. Ran OD150. No problems ---------------------------------------------------- 11 Oct 2009 Markos reran OD148 - no missing packets ran OD149 - 2 events, 4 execution failures, 1505 CHK packets missing re-ran OD149 - 2 events, 4 execution failures, 5 CHK packets missing ---------------------------------------------------- 10 Oct 2009 Sunil Ran OD148 - 52 CHK packets missing Reran ODs 147-148 - Now no missing CHK packets in OD148 ---------------------------------------------------- 09 Oct 2009 Sunil Reran ODs 144-145 - Now no missing CHK packets in OD145 Ran OD147 - no missing packets ---------------------------------------------------- 08 Oct 2009 Andreas run ods 145-146 od 145 has 13 missing critical packets ----------------------------------- 06 Oct 2009 Sunil Reran OD143 - One CHK packet still missing near to the time the SMEC motor current OBCP triggered ran OD144 - Nominal (no TM gaps). SPIRE was in REDY mode in OD144 ------------------------------------- 05 Oct 2009 Ken Reran OD142 - OK ran OD143 - 3 TC execution errors, 1 missing Critical Hsk packet. These occurred during the trigger of the SMEC Current limit OBCP, which reset the instrument to REDY mode and disabled the SPIRE timeline. This happened during the slew to the first target of the OD. Consequently, no observations (apart from the first PCAL flash) were executed for OD 143, and no observations will be executed for OD144. OD145 is being replanned as a SPIRE photometer day ------------------------------------- 04 Oct 2009 Ken reran OD140, ran OD141 and OD 142 - no problems ------------------------------------- 2 Oct 2009 Ed reran OD139 and ran OD140. No problems. ------------------------------------- 1 Oct 2009 Ed reran OD138 and ran OD139. No problems. ------------------------------------- 30 Sept 2009 Ed reran OD137, and ran OD138. No problems. Missing packets from OD137 retrieved. ------------------------------------- 29 Sep 2009 Markos reran OD136 no events, no missing packets ran OD137 25 photometer missing packets ------------------------------------- 28 Sep 2009 Markos reran OD135 no missing packets but 4 events, 8 TC failures, 2 transparent packets. In the briefing it was mentioned that the OD failed ran OD136 no missing packets ------------------------------------- 26-27 Sep 2009 Andreas run 134-135 OD134 - 338 missing CHKT (run twice, second time on OD135) OD134 - 1547 Photometer packets missing OD135 - 4 CHKT missing ------------------------------------ 25 Sep 2009 Sunil reran OD132, no missing packets ran OD133, no missing packets ------------------------------------- 24 Sep Markos reran OD131, no missing packets ran OD132, no missing packets ------------------------------------- 23 Sep Markos reran OD130, no missing packets ran OD131, no missing packets ------------------------------------- 22 Sep Ed ran OD130 even though propagation not complete. No missing packets so far One expected event at beginning of SCAL on (VM stopped) Will run again when propagation complete Repeated - no problems found (VM events now for each SCAL observation) ------------------------------------- 21 Sep 2009 Ken reran OD128 Ran OD129 - no problems ------------------------------------- 20 Sep 2009 Ivan Ran OD128 - no problems ------------------------------------- 19 Sep 2009 Ken Reran OD126 Ran OD127 - no problems ------------------------------------- 18 Sept 2009 Ed Reran HM for OD125 ran OD126 - no problems. ------------------------------------- 17 Sept 2009 Ed Reran HM for OD124 ran OD125 - no problems. Ran some previous early ODs to see if packets recovered - some were recovered OK, but not all. ------------------------------------- 16 Sept 2009 Ed Reran HM for OD123. ran OD124 - no problems - the missing packets from OD123 have appeared. ------------------------------------- 15 Sept 09 Andreas Ran OD123. No Nominal/Critical HK packets missing 49 Spectrometer packets missing. Obslog for 123 properly filled now (wasn't before) ------------------------------------- 14 Sept 09 Ken Re-ran OD121 HM Ran OD122 - 5 errors raised ("VM Stopped") these are expected when stopping the SCAL VM, before restarting it in SpireSpectroScalOn -------------------------------------- Ran OD121 and found no missing packets or events. -------------------------------------- 12 Sept 09 Ed re-ran OD119 and then ran OD120. Not all data recieved yet (propagation complete email not arrived at 9.30am. It was a long OD so only finished at 23.11 last night. No problems in the data recieved so far. -------------------------------------- 11 Sep 09 Markos Re-ran OD118 then ran OD119 No gaps, no missing packets Everything else is in order too -------------------------------------- 10 Sep 09 Andreas Re-run OD116-117-118 Health Mon seems to be in order (No missing CHK packets) Everything else is in order too -------------------------------------- 09 Sept 09 Ed Reran OD116. ran OD117. The 150 missing NHK have appeared. Ken noticed that there were only 112 CHK packets recieved in OD116. There were 113783 CHK packets recieved in OD117 (with 4570 missing) Further investigation showed: - CHK packet total in summary file includes CHK packets + events + commands (explains the large number - there should be only 1 CHK pkt per 2 secs) - CHK packets stop 6 sept 1:39:18 and start again 7 sept 1:39:20 - checked HSC gap checker and they note a gap with same period. - the gap explains why there were no events reported (not even expected events) on OD116 even though spire was observing. -------------------------------------- 08 Sept 09 Ed Reran OD115 & ran OD 116. 150 missing NHK packets but otherwise no problems. -------------------------------------- 07 Sept 2009 Ed Re-ran OD114 Ran all for OD115 - No missing packets or events -------------------------------------- 06 Sept 2009 Ken Reran HM for OD113 Ran all for OD114 - No missing packets or events. -------------------------------------- 05 Sept 2009 Tanya Ran everything for OD113 TC execution failures during OBCP trigger One missing CHK packet also during OBCP trigger -------------------------------------- 04 Sept 2009 Tanya Re-Ran HM for OD111 Ran everything for OD112, no missing packets in 112 FIFO full errors continued for this second parallel day ------------------------------------- 03 Sept 2009 Ed Re-ran HM for OD 110 Ran HM for OD111 (parallel mode) - errors (pool full) at end of every observation - occurs during the last PCAL flash. Ken looking into it. No missing packets. Parallel mode cooler recycle looks OK. Ran future days 116 & 117 -------------------------------------- 02 Sept 2009 Ed Re-ran HM for OD 109 Ran HM, ObsLog & TA for OD 110 - no missing packets or events. Re-ran HM for OD94/95 - missing SuSD packets now recovered. -------------------------------------- 01 September 2009 Ken Ran HM for OD108 Ran Ops Processing for OD109 - No missing packets or events -------------------------------------- 31 August 2009 Ken Ran HM for OD107 Ran Ops Processing for OD108 - No missing packets or events Ed updated HM to new version: HealthMonVer=1.24 -------------------------------------- 30 August 2009 Ken Ran HM for ODs 100 to 107 - this propagated the packet count for photometer science data packets - now no missing packets in OD106 Ran other Ops progs for OD 107 No missing packets or events -------------------------------------- 29 August 2009 Markos Ran OD 104 again no missing packets, no events Ran OD 105 no missing packets, no events Ran OD 106 505 events all cleared, 25 photometer missing packets -------------------------------------- 26 August 2009 Ken Ran OD 104 HM - OK wakefield crashed propagation crashed just before the end -------------------------------------- 25 August 2009 Tanya Ran HM for OD 101 and 102 Ran log for 102 Tanya Ran HM for OD 101 and 102 Ran log for 102 Ran TA for 102 -------------------------------------- 24 August 2009 Ken Ran OD99 Health Monitoring to get rid of OD100 gaps but still gaps from OD98 - also took a long time due to DB backup running Ran OD100 as it had not been run yesterday Ran OD101 Ran OD98 and OD99 health monitoring to clear CHK gaps Two new events, in ODs 98 and 99, occured during Table 50 dumps indicating LS buffer overflow - SPR Spire-1855 Updated HealthMonitoring script to recognise 'Unkown' events. Failed to get them into CVS - SPR Spire-1862 -------------------------------------- 22 Augusut 2009 Andreas Rerun OD99 and everything seems to work fine. Looking at the operational_days.txt there was no entry for od99 even though sara said she run operations for that day. Considering the possibility that Sara is not in the "spire" group. Further investigation required. Sarah Ran OD99 ObsLog Got this error. Have no idea if it should work from my account. 2-Aug-09 10:26:49.032 INFO IndexManager: Begin to convert tags index format 22-Aug-09 10:26:49.151 INFO IndexManager: End to convert tags index format storing SpireOps_ObsLog_DailyLog_0099 found tags: ['SpireOps_ObsLog_DailyLog_0006', 'SpireOps_ObsLog_DailyLog_0011', 'SpireOps_ObsLog_DailyLog_0012', 'SpireOps_ObsLog_DailyLog_0013', 'SpireOps_ObsLog_DailyLog_0044', 'SpireOps_ObsLog_DailyLog_0056', 'SpireOps_ObsLog_DailyLog_0057', 'SpireOps_ObsLog_DailyLog_0058', 'SpireOps_ObsLog_DailyLog_0066', 'SpireOps_ObsLog_DailyLog_0077', 'SpireOps_ObsLog_DailyLog_0099'] Exception in thread "Thread-2" Traceback (innermost last): File "/misc/winchester1/ops/operations/scripts/./ObservationLog.py", line 291, in run File "/misc/winchester1/ops/operations/scripts/./ObservationLog.py", line 372, in combineLogs File "/misc/winchester1/ops/operations/scripts/./ObservationLog.py", line 465, in addLog AttributeError: __getitem__ HealthMon worked ok. TA task ran in 0s. ------------------------------------------ 21 August 2009 Ed Ran OD 98 no missing packets 1 event during REDYtoPHOT_STBY but it was cleared. ------------------------------------------ [20 Aug - Andreas - OD 97] ------------------------------------------ 19 August 2009 Ed Ran HM for OD 95 and then OD96. No missing packets. Trend analysis worked OK (but no spire observations on OD96) ------------------------------------------ 18 August 2009 Ken Ran OD94 then OD95 - health monitoring took ~30 mins! Lots of missing Subsystem Science packets - MOC report many 'Unkown' packets - I suspect that these are PTC diagnostic packets, which should not be being generated during normal observations. See SPR- Many events, but all as expected, including Stop-VM packet (identified as 'unknown') during PtcOff observation Observation Logging had not produced anything after 30 mins so killed the program and tried again - eventually it worked (note backup was running at the time and take 100% of one processor on Chesterfield) Trend Analysis still crashing ------------------------------------------ 17 August 2009 Ken Ran OD93 then OD 94 - lots of missing Subsystem Science packets Many events, but all as expected, including Stop-VM packet (identified as 'unknown') during PtcOff observation Start_ICC_OPs was only allocated 512 Mb - therefore failed during Health Monitoring. JyLaunch properties changed to provide 1.5GB - ran OK Trend Analysis Crashed - SPR 1822 raised ------------------------------------------ 16 August 2009 Markos Ran OD92 which was not propagated yesterday. Got 321 critical packets missing. Ran OD91 again and the re-ran OD92, missing critical packets dissapeared. No events, no missing packets also for OD93 ------------------------------------------ 15 August 2009 Ed Propagation was not completed in the morning. Only 290 housekeeping packets had been propagated. Therefore, need to run OD92 again tomorrow. ------------------------------------------ 14 August 2009 Ken Ran OD91 - no gaps, no error events after running OD90 again ------------------------------------------ 13 August 2009 Ken Ran OD90 - no gaps, no error events after running OD89 again ------------------------------------------ 11 August 2009 Ed Ran OD89 - no gaps after running OD88 again first. ------------------------------------------ 10 August 2009 Ed Ran OD88 - no missing packets after re-running OD87 first. See spike in subKtemp from SMEC being driven into end stop. --------------------------------------------------- 10 August 2009 Ed Ran OD87 - no missing packets after re-running OD86 first. Obslog - error with dateParameter. The reason for the permissions problems was that I used the script RunAllODs_v1.py which does not set permissions! Steve made all FITS files executable. This is not needed so remove this (just need read-write). To fix this, created a new executable file called "start_ICC_Ops_allDays", which calls umask and then the RunAllODs_v1.py script. Also a problem in ObsLog that manual commanded observations no longer logged. Fixed and rerun for comP days. ObsLogVer=1.36 --------------------------------------------------- 09 August 2009: Markos ran OD86 - 1864 events,no missing packets 932 No S/D/M CU Responses, all cleared Obslog is not working: Removing observations that were not actually scheduled Written File /misc/winchester1/ops/operations/logs/OD_0086/SpireOps_ObsLog_DailyLog_0086_v5.csv Written File /misc/winchester1/ops/operations/logs/OD_0086/SpireOps_ObsLog_AorParams_0086_v5.csv storing SpireOps_ObsLog_DailyLog_0086 Exception in thread "Thread-13" Traceback (innermost last): (no code object) at line 0 java.io.IOException: Failed to save product using FitsArchive. File:/misc/winchester1/ops/operations/products/SpireOps_ObsLog/herschel.ia.dataset.Product/SpireOps_ObsLog_DailyLog_0086.fits /misc/winchester1/ops/operations/products/SpireOps_ObsLog/herschel.ia.dataset.Product/SpireOps_ObsLog_DailyLog_0086.fits (Permission denied) at herschel.ia.pal.pool.lstore.BasicDataFileManager.save(BasicDataFileManager.java:141) at herschel.ia.pal.pool.lstore.FitsProductPool.save(FitsProductPool.java:244) at herschel.ia.pal.pool.lstore.FitsProductPool.save(FitsProductPool.java:199) at herschel.ia.pal.pool.lstore.FitsProductPool.save(FitsProductPool.java:859) at herschel.ia.pal.pool.lstore.FitsProductPool.save(FitsProductPool.java:851) at herschel.ia.pal.StoredVersionHandler.save(StoredVersionHandler.java:148) at herschel.ia.pal.ProductStorage.commit(ProductStorage.java:985) at herschel.ia.pal.ProductStorage.save(ProductStorage.java:864) at herschel.ia.pal.ProductStorage.saveAs(ProductStorage.java:918) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java) at org.python.core.PyMethod.__call__(PyMethod.java) at org.python.core.PyObject.__call__(PyObject.java) at org.python.core.PyInstance.invoke(PyInstance.java) at ObservationLog$py.saveProduct$31(/misc/winchester1/ops/operations/scripts/./ObservationLog.py:1219) at ObservationLog$py.call_function(/misc/winchester1/ops/operations/scripts/./ObservationLog.py) at org.python.core.PyTableCode.call(PyTableCode.java) at org.python.core.PyTableCode.call(PyTableCode.java) at org.python.core.PyTableCode.call(PyTableCode.java) at org.python.core.PyFunction.__call__(PyFunction.java) at org.python.core.PyInstance.invoke(PyInstance.java) at ObservationLog$py.run$3(/misc/winchester1/ops/operations/scripts/./ObservationLog.py:290) at ObservationLog$py.call_function(/misc/winchester1/ops/operations/scripts/./ObservationLog.py) at org.python.core.PyTableCode.call(PyTableCode.java) at org.python.core.PyTableCode.call(PyTableCode.java) at org.python.core.PyTableCode.call(PyTableCode.java) at org.python.core.PyFunction.__call__(PyFunction.java) at org.python.core.PyMethod.__call__(PyMethod.java) at org.python.core.PyObject.__call__(PyObject.java) at org.python.core.PyObject._jcallexc(PyObject.java) at org.python.core.PyObject._jcall(PyObject.java) at org.python.proxies.ObservationLog$ObservationLogger$9.run(Unknown Source) Caused by: java.io.FileNotFoundException: /misc/winchester1/ops/operations/products/SpireOps_ObsLog/herschel.ia.dataset.Product/SpireOps_ObsLog_DailyLog_0086.fits (Permission denied) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.(FileOutputStream.java:179) at java.io.FileOutputStream.(FileOutputStream.java:70) at herschel.ia.io.fits.FitsArchive.save(FitsArchive.java:371) at herschel.ia.io.fits.FitsArchive.save(FitsArchive.java:331) at herschel.ia.pal.pool.lstore.BasicDataFileManager.save(BasicDataFileManager.java:139) ... 34 more java.io.IOException: java.io.IOException: Failed to save product using FitsArchive. File:/misc/winchester1/ops/operations/products/SpireOps_ObsLog/herschel.ia.dataset.Product/SpireOps_ObsLog_DailyLog_0086.fits /misc/winchester1/ops/operations/products/SpireOps_ObsLog/herschel.ia.dataset.Product/SpireOps_ObsLog_DailyLog_0086.fits (Permission denied) Steve fixed permissions and it worked --------------------------------------------------- 8 Aug 2009 Sunil Ran OD85 OD logging seemed to work OK but no log files have been put on the web. File permission errors: Removing observations that were not actually scheduled No SPIRE observations were found in that Operational Day ===> writing an empty f ile.. Written File /misc/winchester1/ops/operations/logs/OD_0085/SpireOps_ObsLog_Daily Log_0085_v1.csv Written File /misc/winchester1/ops/operations/logs/OD_0085/SpireOps_ObsLog_AorPa rams_0085_v1.csv 08-Aug-09 09:37:01.248 INFO HandlerFactory: Version strategy is track 08-Aug-09 09:37:01.252 INFO PoolCreatorFactory: Creating pool SpireOps_ObsLog of type lstore from defaults storing SpireOps_ObsLog_DailyLog_0085 Exception in thread "Thread-2" Traceback (innermost last): (no code object) at line 0 java.io.IOException: Failed to save product using FitsArchive. File:/misc/winche ster1/ops/operations/products/SpireOps_ObsLog/herschel.ia.dataset.Product/SpireO ps_ObsLog_DailyLog_0085.fits /misc/winchester1/ops/operations/products/SpireOps_ ObsLog/herschel.ia.dataset.Product/SpireOps_ObsLog_DailyLog_0085.fits (Permissio n denied) at herschel.ia.pal.pool.lstore.BasicDataFileManager.save(BasicDataFileMa nager.java:141) at herschel.ia.pal.pool.lstore.FitsProductPool.save(FitsProductPool.java :244) at herschel.ia.pal.pool.lstore.FitsProductPool.save(FitsProductPool.java :199) at herschel.ia.pal.pool.lstore.FitsProductPool.save(FitsProductPool.java :859) at herschel.ia.pal.pool.lstore.FitsProductPool.save(FitsProductPool.java :851) at herschel.ia.pal.StoredVersionHandler.save(StoredVersionHandler.java:1 48) at herschel.ia.pal.ProductStorage.commit(ProductStorage.java:985) at herschel.ia.pal.ProductStorage.save(ProductStorage.java:864) at herschel.ia.pal.ProductStorage.saveAs(ProductStorage.java:918) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java ) at org.python.core.PyMethod.__call__(PyMethod.java) at org.python.core.PyObject.__call__(PyObject.java) at org.python.core.PyInstance.invoke(PyInstance.java) at ObservationLog$py.saveProduct$31(/misc/winchester1/ops/operations/scr ipts/./ObservationLog.py:1219) at ObservationLog$py.call_function(/misc/winchester1/ops/operations/scri pts/./ObservationLog.py) at org.python.core.PyTableCode.call(PyTableCode.java) at org.python.core.PyTableCode.call(PyTableCode.java) at org.python.core.PyTableCode.call(PyTableCode.java) at org.python.core.PyFunction.__call__(PyFunction.java) at org.python.core.PyInstance.invoke(PyInstance.java) at ObservationLog$py.run$3(/misc/winchester1/ops/operations/scripts/./Ob servationLog.py:290) at ObservationLog$py.call_function(/misc/winchester1/ops/operations/scri pts/./ObservationLog.py) at org.python.core.PyTableCode.call(PyTableCode.java) at org.python.core.PyTableCode.call(PyTableCode.java) at org.python.core.PyTableCode.call(PyTableCode.java) at org.python.core.PyFunction.__call__(PyFunction.java) at org.python.core.PyMethod.__call__(PyMethod.java) at org.python.core.PyObject.__call__(PyObject.java) at org.python.core.PyObject._jcallexc(PyObject.java) at org.python.core.PyObject._jcall(PyObject.java) at org.python.proxies.ObservationLog$ObservationLogger$9.run(Unknown Source) Caused by: java.io.FileNotFoundException: /misc/winchester1/ops/operations/products/SpireOps_ObsLog/herschel.ia.dataset.Product/SpireOps_ObsLog_DailyLog_0085.fits (Permission denied) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.(FileOutputStream.java:179) at java.io.FileOutputStream.(FileOutputStream.java:70) at herschel.ia.io.fits.FitsArchive.save(FitsArchive.java:371) at herschel.ia.io.fits.FitsArchive.save(FitsArchive.java:331) at herschel.ia.pal.pool.lstore.BasicDataFileManager.save(BasicDataFileManager.java:139) ... 34 more java.io.IOException: java.io.IOException: Failed to save product using FitsArchive. File:/misc/winchester1/ops/operations/products/SpireOps_ObsLog/herschel.ia.dataset.Product/SpireOps_ObsLog_DailyLog_0085.fits /misc/winchester1/ops/operations/products/SpireOps_ObsLog/herschel.ia.dataset.Product/SpireOps_ObsLog_DailyLog_0085.fits (Permission denied) ------------------------------ 7 Aug 2009 Ed Ran OD84. updated cvs versions: ObsLogVer=1.34 add column with decimal degrees for RA/Dec HealthMonVer=1.23 speed up SSC calculation StartGuiVer=1.3 remove memory allocation option Re-run all OD for obs Log to add the new columns. Re-run HM to get rid of gaps in the last few days Ran future logs for OD88/89 ------------------------------ 6 Aug 2009: Sunil ran OD83 135 CHK packets missing re-ran OD81-82 - 58 CHK packets still missing from OD81. ------------------------------ 4-5 Aug 2009: Andreas ran OD81-82 Critical missing packets at start of OD82 You might have to rerun OD81-82 Health monitoring to see if packets arrived. ------------------------------ 03 August 2009: Markos ran OD80 - no events, 25 critical missing packets ran 0D79 and then OD80, missing packets dissapeared --------------------------------------------------- 02 August 2009: Markos ran OD79 - no events or missing packets --------------------------------------------------- 01 August 2009: Ken ran OD78 - no events or missing packets --------------------------------------------------- 31 July 2009: Sunil ran OD77 - everything is nominal. 2 unknown events reported by HM. But these are the expected Peakup event reports from the two Peakup mode test observations on Alpha Boo. The requested Y and Z offsets in the two cases are (0.04,0.03) and (0.04,0.04) arcseconds respectively. Now we to need to optimise the chop and jiggle parameters to the Peakup command. HM should be updated to include these events. SXR to be raised. 24 Photometer science packets missing GAP 30-07-2009 12:54:08 UTC 30-07-2009 12:54:09 UTC 6907 6930 24 Ed updated the obsLog to v1.31 to correct for problem in the calculation of RA and to separate the future logs in the table - re-ran for all the relevant SPIRE days. - ran the future log for ODs 82 and 83. --------------------------------------------------- 30 July 2009: Sunil ran OD76 - processing was extremely slow, taking over 10 minutes each for OL, HM and TA. - no unknown events or missing packets. re-ran HM for OD71 - the 11 missing PSD packets have been recovered. --------------------------------------------------- 29 July 2009: Ken ran OD75 - no events or missing packets --------------------------------------------------- 28 July 2009: Ed updated the HealthMonitoring version to 1.21 --------------------------------------------------- 28 July 2009: Ken ran OD74 - no events or missing packets --------------------------------------------------- 27 July 2009 Ed Update ObsLogVer = 1.30 - fix to declination re-run HM for OD 71 to see if missing phot packets appeared - 11 pkts still missing re-run HM for the previous days that had gaps to see if they have been recovered. --------------------------------------------------------------------------- 21-25 Jul 2009 Andreas Run ODs: 67-72 OD71:11 Missing science packets. --------------------------------------- 20 July 2009 Markos No missing packets. 254 events:127 NO *CU responses and 127 clearances of the NO responses Update CVS version of ObsLog to ObsLogVer=1.28 subsequently to ObsLogVer=1.29 Update CVS version of HealthMon to Ver=1.20 --------------------------------------------------------------------------- 19 July 2009 Markos Only 2 missing CHK in OD65 No events, no other missing packets Re-ran OD 64 and then ran OD 65. Missing packets gone. --------------------------------------------------------------------------- 18 July 2009 Ed A few missing CHK in OD64, so re-ran OD 63 and then ran OD 64. Missing packets gone. There are missing NHK on OD 58. Re-ran OD57, and then OD58 and the missing packets appear. Update CVS version of ObsLog to ObsLogVer=1.27 This changes the web site (only last 30 days shown and give a link to AOR parameters). It also allows processing future days and makes a log if it finds a observations that have been scheduled. --------------------------------------------------------------------------- 17 July 2009 Sunil A "propagation completed" email message was not received from the HSC for OD63. During the videocon HSC stated that the propagation was completed successfully but the lack a message was apparently caused by a TM ingestion bug. First re-ran OD62 - NHK gaps gone but the CHK gaps still present. Re-ran OD61 and OD62. Now the CHK gaps in OD62 have also one. Ran OD63 Everything nominal No packets missing --------------------------------------------------------------------------- 16 July 2009 Sunil Ran OD62 3000 missing NHK packets 368 missing CHK packets --------------------------------------------------------------------------- 15 July 2009 Ed Ran OD 61. Some missing CHK packets. Re-ran OD 60. The missing packets were recovered when running OD 61 for the second time. --------------------------------------------------------------------------- 14 July 2009 Ed Ran health monitoring for OD60. Found some missing CHK packets but time was starting 10 minutes before beginning of the OD. Re-ran OD 59, and then re-ran OD60 again, and the missing CHK were recovered. Also re-ran OD 56 to check whether more SuSD packets were recieved but no change since yesterday. --------------------------------------------------------------------------- 13 July 2009 Ed re-ran from OD56. A few of the SuSD packets have been retrieved but still many missing on OD56 Missing NHK from OD 57 are retrieved Missing CHK and spec packets from OD 58 retrieved but still missing NHK No missing packets for OD 59 --------------------------------------------------------------------------- 12 July 2009 Tanya Propagation message received, quire a few missing packets some spectrometer science --------------------------------------------------------------------------- 11 July 2009 Ed Re-ran monitoring for ODs 53-57 the missing CHK from OD55 have now appeared. the missing CHK from OD56 have now appeared BUT, there are now missing subsystem science packets from OD 56 (19925 missing packets) There are also transparent packets and associated events indicating PHOTF Frame and BSM Frame Checksum errors - need further investigation. There are 150 missing NHK packets from OD 57 HSC confirmed the missing SuSD packets - it seems an issue between HSC and MOC. ----------------------------------------------------------- 10 July 2009 Sunil "Propagation completed" message received at 03:14BST Ran for OD56 198 CHK packets missing No other packets missing ----------------------------------------------------------- 9 July 2009 Ed Ran for OD 55 1 hour of missing NHK. Larry seemed to know about it and was going to chase up MOC. Re-ran health monitoring at 5.30 and the NHK gap had gone. Still some missing CHK at beginning of day ----------------------------------------------------------- 8 July 2009 Ed Ran for OD54 - propagation not complete until about 11am because it was stopped yesterday and only restarted this morning No missing packets or events Re-ran obsLog for ODs 49-53 because the manual commanded names had not appeared properly in the versions from yesterday. ----------------------------------------------------------- 7 July 2009 Markos ran for OD53 no events no packets missing ----------------------------------------------------------- 6 July 2009 Markos ran for OD52 CHK and NKK packets missing new version of the obsLog: ObsLogVer=1.26 re-ran obslog for all OD06-52 ----------------------------------------------------------- 5 July 2009 Markos/ken ran for OD50/51 CHK packets missing 44 events, they are been checked ----------------------------------------------------------- 30 June 2009 Markos ran for OD46 NHK and CHK packets missing No events ----------------------------------------------------------- 29 June 2009 Tanya ran for OD45 6 minutes of CHK missing at end of day, otherwise OK ----------------------------------------------------------- 28 June 2009 Ed ran for OD 44. Some missing CHK packets. Check that all the data were here for yesterday - re-ran health monitoring for OD 42-44 -> most of the gaps disappear (except for the missing subsystem science packets on OD42. ----------------------------------------------------------- 27 June 2009 Ed Ran scripts for OD 43. Propogation email hadn't come but in daily briefing they said that there was nothing more in the queue for spire just they had a problem with sending the complete message. Therefore we should have everything. Not sure why there are subsystem science packets, when there were no spire operations in OD 43. ------------------------------------------------------------------------------------------- 26 June 2009 Ed Ran scripts for OD 42 new version of the obsLog: ObsLogVer=1.24 Re-ran for all Spire ODs. I compiled all of the files which had the obsMode for manual commanded observations into a single file: manualCommandedObservations.txt The observations logger reads this file and fills in the names. Removed the old files from the web page. Also updated so that manual commanded observations have the correct end time. --------------------------------------------------------------------------------------- 25 June 2009 Sunil Ran scripts for OD 41. No problems - no missing packets. Instrument in REDY mode. Note that the HSC Gap Checker seems to find the following gap: +---------------------------+ | Apid 1282 | +--------------------------------------------------------------------------------------+ | Type | Miss SSC | # SSC | Start time to Last time | +--------------------------------------------------------------------------------------+ | Gaps | 13351 | +1050 | 2009-06-23T21:19:56.756454 to 2009-06-23T22:30:00.726959 | +--------------------------------------------------------------------------------------+ But these gaps are definitely are not present in our DB. I have checked this by plotting the NHK_SSC parameter for this time period. ----------------------------------------------------------------------------------------- 24 June 2009 Ed Somehow we have got out of sync by one day: ran scripts for ODs 39 and 40. No problems - no missing packets. The times of the ODs have been changing in the last few days: some ODs longer than 24 hrs some shorter. Apparently this is correct because the time of DTCP has been changing. The start of DTCP fixes the start of that OD. The times of DTCP can be checked here: http://www.herschel.be/twiki/bin/view/HSC/HSCMissionPlanningCOP ----------------------------------------------------------------------------------------- 23 June 2009 Markos Ran scripts for OD 38 CHK gaps ----------------------------------------------------------------------------------------- 22 June 2009 Ken Ran scripts for OD 37 No problems (instrument in REDY) CHK gaps at start of OD , as usual ----------------------------------------------------------------------------------------- 21 June 2009 Ken Ran scripts for OD36 to see if missing data appeared Actually gaps identified are there because the instrument was switched on during this OD The NHK Source Sequence Count starts from 0. The CHK sourcesequence count starts from where the boot software finishes - in this case 19 4 TC Execution failures - Table Reports for tables not present in the OBS image ----------------------------------------------------------------------------------------- 20 June 2009 Markos Ran scripts for OD35. There is no SPIRE data because the instrument is switched off. Ran scripts for OD36. 3 Events: All expected 4 TC Execution Failures: 18-06-2009 11:41:00 UTC 0d00 c108 001b 0001 0800 60cc 866e 23b3 1d00 e1cc 0811 000f 0908 0400 1185 003c 99a2 18-06-2009 11:41:04 UTC 0d00 c10c 001b 0001 0800 60cc 8672 23b4 1d00 e1cd 0811 000f 0908 0400 ab36 003d 9b1c 18-06-2009 11:41:37 UTC 0d00 c12e 001b 0001 0800 60cc 8693 23ad 1d00 e1d6 0811 000f 0908 0400 facc 0066 ffec 18-06-2009 11:41:41 UTC 0d00 c132 001b 0001 0800 60cc 8697 23ac 1d00 e1d7 0811 000f 0908 0400 407f 0067 3725 Explanation: REPORT_TABLE command failures in the beginning of the procedure (DPU_START_SP) are due to the fact that these tables were not in the OBS image in the EEPROM (see also OD36 log by Markos & Sunil COP_Log_OD36.doc) Missing NHK and CHK packets (Played a little bit with the GUI and added OD37, OD38 which are due to be studied on Sunday, Monday, although there were some NHK packets for OD37 due to a level of propagation) ----------------------------------------------------------------------------------------- 18 June 2009 Ed Ran scripts for OD34. There is no SPIRE data because the instrument is switched off. New version of ObsLog with the first column showing the operational day number: ObsLogVer=1.23 Re-run all ODs so that the OD number goes into the combined log in the first column. Added a web page to list known issues with the data - this is in the web_reports/ directory (same place as the latest news page) ----------------------------------------------------------------------------------------- 17 June 2009 Markos Run scripts for OD 33 1 event: Cmd 8c3f0011 750 Nominal HK packets missing --------------------------------------------------------------------------------------------------- 15 June 2009 Ken Ran scripts for OD31 Instrument Health: Nominal Approximately 1050 missing NHK packets (About 70mins) Propagation completed last night Bulk transfer failed at 13:30 yesterday - manually restarted - ran OK Bulk transfer at 02:00 last night was cancelled (because of previous problems) - Huw started this morning - running OK 14 June 2009 Ken Ran scripts for OD30 No unexpected events Photometer and subsystem science data missing for start of OD 10 mins od critical hsk data missing Yesterday's command failure was seen by Sunil at the time, during the procedure to uplink new tables. It is possible due to a command to stop a VM which was already stopped, or to clear a table, which was already empty - but this should be followed up in case a table has not been uploaded correctly (again) - I need to speak with Sunil about this A parameter event failure was also found during this procedure (PUMPHTRTEMP out of limits) but this is expected. 13 June 2009 (Andreas: I think Tanya meant OD 29 :-) - Re-run TA for 28-29 to produces plots with appropriate scale) Tanya Ran scripts for OD30 The usual missing packets were observed One command failure was reported but nothing obvious at that time in the events log. 12 June 2009 Tanya Ran scripts for OD28 11 June 2009 Markos Run scripts for OD 27 14 events received. 7 were no responses and were cleared (7) Critical and spectrometer science packets missing --------------------------------------------------------------------------------------------------- 10 June 2009 Markos Run scripts for OD 26 4 events received. 1 was no SCU response, 1 was no DCU response both cleared. Critical, photometer science and SCU packets missing New CVS version ObsLogVer=1.22 Re-ran obslog script for OD 26 which has now the pointing observations added Re-ran healthmon for all OD 6-26 to check for missing science packets Re-ran obslog for all OD 6-26 to check whether there were pointed observations that were missed --------------------------------------------------------------------------------------------------- 09 June 2009 Ed Run scripts fpr OD 25 Some missing NHK packets at beginning of day. --------------------------------------------------------------------------------------------------- 08 June 2009 Ed Run scripts for OD 24 Some NHK gaps. These are the same as in the HSC gap checker. --------------------------------------------------------------------------------------------------- 07 June 2009 Ed Run scripts for OD 23. Fix ObservationLog script for an error in the value it puts for RA/Dec when no pointing And to fix for OD 6-10 which have wrong times attatched to the observation object. Now the script can be run for all ODs and will do the correct thing for OD 6-10. Luca had asked a while ago why there was no observation 0x5000000C5 in the log for OD11. Checked and this is because it corresponds to a period with missing NHK packets. New CVS version ObsLogVer=1.21 Re-ran observation log for all previous days (this removes the HIFI and PACS observations that were in some days). --------------------------------------------------------------------------------------------------- 06 Jun 2009 Run scripts for OD 22, No popup when ran health monitoring, procedure indicated there should be Tried again after shutting down gui, this time it worked The procedure for running the HSC gap checker is unclear I tried 'source getLatestHscGapRep' but am not sure if that was the right things to do Anyway a file was produced but showed a lot of gaps 05 Jun 2009 Run Scripts for od 21 No Events, No Missing packets. Andreas 04 June 2009 Run scripts for od 20 150 NHK packets missing, Also missing in HSC gap checker 750 CHK packets missing (although they correspond to the end of OD19) 03 June 2009 Markos Ken changed the Health Mon Script. New CVS version: HealthMonVer=1.19 Ran new Health Mon Script for OD6-18 OD6: 4 events (SCU, DCU, MCU response cleared, ERROR: Cmd:a8e00000) gaps is Critical packets and Subsystem science data OD7: No events, no gaps OD8: No events, gaps in Spectrometer science and subsystem data OD9: No events, gaps in Critical packets OD10: 286 events all MCU/DCU/SCU responses which were leared, gaps in NHK OD11: 38 events all cleared DCU responses, gaps in NHK and Photometer science data OD12: No events, no gaps OD13: No events, gaps in Critical packets OD14: No events, gaps in photometer science and subsystem data OD15: No events, no gaps OD16: No events, no gaps OD17: 6 events, only MCU no responses, all cleared , no gaps OD18: No events, no gaps Ran HealthMon, TA,ObsLog scripts for OD19 No Gaps, no Events --------------------------------------------------------------------------------------- 02 June 2009 Markos Ran scripts for OD 18 No Gaps, no events Problem with missing gaps identified both in our Health Mon Script and HSC gap checker. --------------------------------------------------------------------------------------- 01 June 2009 Markos Ran scripts for OD 17 6 events, 3 MCU no responses, all 3 cleared nominal --------------------------------------------------------------------------------------- 31 May 2009 Ed Ran scripts for OD 16 No Gaps, no events --------------------------------------------------------------------------------------- 30 May 2009 Ed Ignore HIFI and PACS observations in the log New CVS version: ObsLogVer=1.19 Ran scripts for OD15. Everything OK. Look at HSC gap checker. See gap in Apid 1284 (PSD) for OD 14. But this is not in our logs. Re-run Health Monitoring for OD14 just to check. Do not see any missing packets in our logs. Put up plots of the second cooler recycle on the latest news page. --------------------------------------------------------------------------------------- 29 May 2009 Ed Run scripts for OD 14 No missing packets No events saw the cooler recycle in TA plots --------------------------------------------------------------------------------------- 28 May 2009 Andreas Run scripts for OD13 No Events 300 CHK packets missing, the same as in the HSA Gap checker log --------------------------------------------------------------------------------------- 27 May 2009 Andreas Run scripts for OD12 No missong packets No events ... Ken made a small change to the Health Mon Script. New CVS version: HealthMonVer=1.14 More updates by Ken New CVS version: HealthMonVer=1.15 More updates by Ken (spotted a bug that was causing HealthMon to be very slow) New CVS version: HealthMonVer=1.16 Ken's now working on HealthMon. I think he figured out how to solve the "missing packets at the start/end day" problem --------------------------------------------------------------------------------------- 26 May 2009 Ed Ran scripts OK. Lots of VM events Problem with order of ODs in Health Monitor web page - fix by sorting the directories. New CVS version: HealthMonVer=1.14 Some NHK gaps - but summary shows a negative number again. OD 10 bulk transfer products appeared in OD 9 directory - raised SPR - for steve to look at tomorrow. Pasquale found a problem with the bias frequency calculation in the observation logs. Corrected. New CVS version: ObsLogVer=1.17 re-ran obsLog for all previous days. Note OD6-10 were using the modified ObservationLog.py to correct for the time shift. OD 11 was run with the proper version from CVS. --------------------------------------------------------------------------------------- 25 May 2009 Ed and Andreas - Health Monitoring script crashed because I had misunderstood the changes needed to the eventDescription() method that Ken sent on 23rd. Updated script and now it works. New CVS version: HealthMonVer=1.13 Missing NHK packets: 23-05-2009 11:39:24 UTC 151 This agrees with the HSC gap checker: +--------------------------+ | Apid 1282 | +-------------------------------------------------------------------------------------+ | Type | SSC Fst | SSC Lst | Start time to Last time | +-------------------------------------------------------------------------------------+ | Gaps | 9411 | 9562 | 2009-05-23T11:39:59.361603 to 2009-05-23T11:50:03.355408 | +-------------------------------------------------------------------------------------+ Observation log found all of the timeline observations OK, but has labelled them as "manually commanded" -> this is because they were swapped from OD 9 to OD 10. The times attatched to the observation objects are still from OD9, so the search using time range for OD 10 does not find any observations. Therefore they are picked up by searching the packets, and my script assumes this means they are manual observations. That means the obsMode is not filled in in the log. Create a txt file in the same format as Luca's file for OD6-9. Trend Analysis: we see when the cooler ran out because the subktemp jumps up. We used the buttons at the bottom of the GUI to plot ODs 8 to 10 to see how long the cooler lasted. --------------------------------------------------------------------------------------- 24 May 2009 Ed and Andreas CHK gaps: 22-05-2009 12:59:23 UTC 366 at 10am meeting: there is some problem at MOC that is causing all of the gaps (incl gaps in spacecraft telemetry) until this is sorted we won't be able to sort the gaps. propagation will be interupted today whilst they backup their system before ingesting the telemetry from OD10 - these are the MTL observations that were put forward by 24 hours. Luca sent a list of OBSIDs and test names - will put up onto the web. Checked the HSC gap results and compared with our results. The 366 missing CHK packets are also found in their logs. --------------------------+ | Apid 1280 | +-------------------------------------------------------------------------------------+ | Type | SSC Fst | SSC Lst | Start time to Last time | +-------------------------------------------------------------------------------------+ | Gaps | 10390 | 10823 | 2009-05-19T15:49:58.978851 to 2009-05-19T16:00:00.970169 | | Gaps | 11326 | 12309 | 2009-05-19T16:09:58.961624 to 2009-05-19T16:30:00.944656 | | Gaps | 13509 | 13875 | 2009-05-22T12:59:58.427567 to 2009-05-22T13:10:00.421463 | +-------------------------------------------------------------------------------------+ Andreas wrote a new script to automatically get the HSC gap checker results. This is in /disks/winchester1/ops/operations/data/getLatestHscGapRep It fetches the gap checker results for the current date. Fixes to the observation logging script: - obsid in hex formatted with 8 characters - add obsid in decimal New CVS version: ObsLogVer=1.17 To fix the time problems which affect OD 6 and OD 7, will need to use a specially modified version of ObservationLog.py. The change is on line 138 to put: observationList = [] This forces the routine to search for observations in the packets rather than using the times given by ObservationAccess. Re-run the other ODs aswell because there is now an extra column with the decimal obsid. at 17.50 they still had not restarted propagation (they had stoped to check the ingestion of OD10) Therefore there is not much seen when running the script now for OD10. --------------------------------------------------------------------------------------- 23 May 2009 Ed & Andreas - created new directory for HSC gap checker results: /disks/winchester1/ops/operations/data/hsc_gap_checker/ Implement change from Ken in Health mon for SID 5113 Correction in ObsLog to search for manual commanded observations after the last timeline observation Correction in ObsLog for getting the bias for spec loadcurves. New CVS versions: ObsLogVer=1.15 HealthMonVer=1.12 --------------------------------------------------------------------------------------- 22 May 2009 Andreas Got gap checker results from HSC. Ken compared for APID 1280 with our results. Found the same gaps in their log -> must be problem from MOC to HSC. Added DPUTEMP to the TA. New CVS version: TrendAnalysisVer=1.17 SpireComonVer=1.16 --------------------------------------------------------------------------------------- 21 May 2009 Ed & Andreas Propagation email was received. Deleted all partial results from tests yesterday in the logs and products directories. All changes to scripts from yesterday uploaded to CVS. Update scripts directory Initial CVS versions used: SpireComonVer=1.15 SpireGuiVer=1.25 TrendAnalysisVer=1.16 TrendAnalysisLvl1Ver=1.17 CustomPlotVer=1.1 ObsLogVer=1.14 HealthMonVer=1.11 StartGuiVer=1.2 GlitchJarVer=1.1 Run HM, OL & TA for OD 6: - events: Kens list had some events that are marked as ERR in our list marked as OK. Ken had more events listed which we did not have - transparent packets: we got 10, but Ken got 15 - the ones at 16:24:31-UTC were not in our report - TC execution failures: 3 - Critical packets - it appears that some are missing but the number in the summary is negative from the detailed report there are high numbers at: 15.49 16.09 17.34 Otherwise there do not appear to be any missing packets. Trend analysis plots look OK. Observations: not sure whether the OBSID reported in the log is the correct one. It seemed that maybe the OBSID in the packet is different to the one attatched to the observation. Need to check whether the list looks as expected. Checked OD 7: there are 9 hours of NHK data already present in the database. Daily videocon: - Asier ran the SPG on OD6 this morning so level 0.5 products should be available to transfer. Steve started bulk transfer. Checked in QLA: for APID 1280 we find 51293 packets. This matches with the number of critical packets in the summary file: > Critical Packets = -14970 of 51293 That indicates something is wrong with the counting of missing packets and actually we don't miss any. Ken sent the scripts that he used. (/home/spire/etp87/Monitor_06.py and /home/spire/etp87/GetTransparentPackets.py) Ran these and compare with HM here: - transparent packets: gets all 15 whereas HealthMon script gets only 10 -> must be something wrong with HealthMon script here... - events - Kens script gives same result as HealthMon - there are missing events: 19-05-2009 15:48:49 UTC MODE (REDY) STEP 0000 0d00 e881 001d 0005 0100 60a5 3383 23ad 0501 5100 0000 0000 8003 0001 0132 0200 0000 d7eb 19-05-2009 16:02:10 UTC MODE (REDY) STEP 0000 0d00 ea9c 001d 0005 0100 60a5 36a4 23ad 0501 5100 0000 200d 8000 0000 0145 0200 0000 c509 i.e. missing events between 0132 and 0145 Looked in CHK_Gaps report: there are indeed missing packets at right time: 19-05-2009 15:49:24 UTC 433 Conclusion: we are missing these packets at RAL but they are present at MOC. Obsids: The times of obsids are 2 hours too early. Ken talked to Luca. They think this is due to assigning CET to the observation objects rather than UTC. Luca will contact Larry. --------------------------------------------------------------------------------------- 20 May 2009 Ed 9.25 Sarah asked Larry: DTCP is at 10.55 zulu Steve: we should get data within a certain time of end of DTCP There is 1.5 hours of data from OD 5 already in there. Propagation completed email received at 02.23. That was for OD 5. Ran Health Monitoring, OD logging and TA. We only have 4575 NHK packets in the database so far. No events or CHK. Steve thought we should have received these before NHK. Asked Larry in 10am videocon. Sounded from him like we should not have anything yet. They are still ingesting uplink info at HSC. Should follow up why we got data before expected. Fixed bugs in Health Monitoring (to do with web page) and OD logging (to do with manual commanded obsns). Trend Analysis worked OK: 1.35 hours of data so far. Ken has sent results for Events and Transparent packets from MOC. Should compare our results with these when we get data. --------------------------------------------------------------------------------------- 19 May 2009 Ed & Andreas: - updated the hcss.props in hcssbld account to contain the operational database: ops_1 - created a script to read specific versions from the CVS - set up the operational direcories