=================================================================== VISTA IOT 2011-06-08, 15-16:30 Garching, 14-15:30 UK, 9-10:30 Chile =================================================================== Minutes (prepared by Valentin D. Ivanov) Participants: Garching: Thomas Szeifert (TSz), Marina Rejkuba (MRe), Wolfgang Hummel (WHu), Michael Hilker (MHi), Magda Arnaboldi (MAr), Gaitee Hussain (GHu), Valentin D. Ivanov (VDI) Paranal: Steffen Mieske (SMi) Via phone: Jim Emerson (JEm), Mike Irwin (MIr) ************************************************************ Preliminary Agenda for the VISTA IOT on Jan 27, 2011 ==================================================== 1. VISTA general post-intervention status and planned activities (TSz) 2. Results from the E2E Survey Operations Review (VDI, MRe) 3. Status of the surveys (MRe, MHi) 4. Phase 3; are there any lessons we learn how to operate better VIRCAM@VISTA as a result from the Phase 3? (MAr) 5. Status of various software: SADT/OT/P2PP/etc (MRe, MHi, JEm) 6. New/recent QC issues, data reduction/product issues (WHu, MIr) 7. Correcting measures for improving the survey efficiency: splitting the LOCS exposures - is it necessary/justified (JEm, Tsz); other measures, if proposed 8. Action Items review (enclosed bellow is the AI section from the minutes of the last IOT) 9. Other issues 10. Date of the next meeting, proposed in the interval Aug 20-30, 2011 ************************************************************ ======================================================================= 1. VISTA general post-intervention status and planned activities (TSz) ======================================================================= ======================================================================= 2. Results from the E2E Survey Operations Review (VDI, MRe) ======================================================================= ======================================================================= 7. Correcting measures for improving the survey efficiency: splitting the LOCS exposures - is it necessary/justified (JEm, Tsz); other measures, if proposed ======================================================================= TSz: VISTA is back in operation, some improvements were implemented; we are back to the original performance with image quality improvements due to AO; the ZP are worse than before, as expected from the aluminum mirrors; no changes of overheads as last defined; however, some improvements in the parallelisation can still be done gaining us more than the estimate of VDI who expects a few minutes per night; other improvement are also possible but have to be confirmed by the engineers, i.e. to speed up the azimuthal axis to shorten the telescope preset. MRe: the survey operations are running well, but we are behind by ~860 hr - this is how much carry over OBs we have; today we calculated the completed observing time during the first 14.5 months, but effectively this is 12 hr because of the interventions and the period when we had images with high ellipticity; so, the completed time is ~1700 hr; the real available time for science was ~2100 hr (this comes from 9.5 hr per night minus 450 hr weather loss yearly, minus 150 hr per year for technical losses and another 150 hr spent on taking standards. Overall, we completed ~1700 out ~2100 hr, i.e. ~63% efficiency. Another way to look at this is from the completion rate of the surveys for the 1st year - the average completion rate is ~15% implying 7-8 yrs to complete the surveys on average. So either the survey area or depth need to be sacrificed if we want to complete them in 5 yrs. VDI: the dry runs yielded the same efficiency of ~65%. MHi: is the time spent to repeat OBs included in the 35% loss? MRe: it is in those 35%, together with reclassified OB, and any extra time loss that went unreported in the night logs MAr: This 65% is not bad - it is the same as efficiency of MegaCam at CFHT after much longer operation and improvements TSz: the next semester we will do better because of the small improvements mentioned before; But the delay is not going to go away completely Re: the 35% also has some of the time lost because of the lower overheads that we charged during P87 SMi: what about the weather loss? MRe: the 35% contain OBs that were aborted in the middle because of worsening weather. Now that we know how slower we are with respect to the expectations, we should look at how to improve our efficiency. As TSz said, some possibilities are as catching/processing errors, longer AO expiration time - these were implemented. Are there any more improvement possible? To gain even 5% is a significant impact because it is equivalent to almost 1/2 yr over the ~8 yr actual duration of the surveys. TSz: we will never achieve what was promised at the beginning an used in the SMPs MRe: this is clear MAr: We should write an internal memo to F. Comeron, F. Patat, B. Leib. A. Kaufer and possibly the DG, to keep this into account for scheduling open time, so the open time program don't compete directly with the surveys; to reinforce this information and to pass it to the upper management; to be done this week, fast enough to save at leas 1 yr which will be lost if we wait until Oct for the PSP. MRe: I already gave Gaitee the info about the carried over OBs: RA range, weather constraints, etc. VDI: will OPO block requests for submitting too many OBs? MAr: yes, if the DG says so TSz: this semester is particularly bad (because of the pending OBs and the intervention) VDI: any crazy ideas are welcomed. What about doing Z band with VST? MRe: let's wait until VST works before even rising the issue TSz: yes VDI: what about splitting the AO exposure to be obtained in parts taken during different offsets? This will save time but it will also improve the image quality, so we can integrate shorted with the science detectors for the same S/N. TSz: too complicated, it will need major revamp of the software MRe: we looked with VDI at the weather losses, they are always a few percent higher than for the UT which rises the question if can we operate with relaxed wind speed limits? TSz: the wind speed closing limit is set because of the dust, the pointing limit is due to some vibration that was confirmed to have been observed by the TIOs during nights with high wind speed VDI: what about increasing the detector's read speed, like for HAWKI? TSz: it is not possible VDI: any comment from JEm? JEm: there is no realistic possibility to reduce the detector overhead, we experimented earlier TSz: the file handling may be a different thing, though JEm: the time to write the file on a disk was speeded up a but by using a faster machine; we saved a little bit on that but certainly the readout time can not be shortened TSz: I am still not convinced about the control loop parameters for the azimuth axis - they appears to be a semi-randomly set optimized to remove the oscillations; I would like to investigate it again. Plus, the parallelisation. VDI: JEm, any suggestions on this option from your side? JEm: we always tried to improve the palatalization but we were afraid to have conflicts between different actions; there might be some reserve there but small. About the AO correction integrating over a few atmospheric cycles when the jitters are shorter - we had put a change request but the modifications were complicated as TSz mentioned, and it was never implemented; it could be done but it is not a quick thing MRe: typically, we need ~45 sec between two jitters to carry out an AO correction, but the AO integration itself is ~30 sec. Let's assume we have enough counts in say 20 sec but there is no enough time to apply the correction before the end of the jitter; Can I keep the exposure and apply it during the next jitter (which is close on the sky so it is unlikely to change the AO correction much)? JEm: we have to wait to integrate over certain time, even if the signal is enough (i.e. the integration can't be shortened ???????) MRe: ????????? JEm: the idea is to integrate 10 sec, then again later for 10 sec, etc. and correct eventually 10 min later (if necessary ????) VDI: so the bottom line is this is hard, and it won't save as much time TSz: in fact, originally we planned to run VISTA in open loop JLe: JEm, presumably you did some back of the envelope calculation how much time it will save, before submitting this request JEm: we were not thinking of tie saving (????) MRe: it will save 20 min per night, so about 1/2 yr for the duration of the public surveys TSz: it will take time to commission it MRe: sure TSz: it is not on the agenda if something working ???????? JEm: Stephen Beard suggested a way for testing in his e-mail TSz: ?????? loosing time JEm: But if I understand, Valentin is suggesting only to identify the best application of resources MRe: another issue is that there are gaps between OBs, usually of 1-4 min; this is possible source of time saving VDI: very likely this time is spent on filling the night log TSz: or hopefully it was used to check the data quality; the best time saving solution is to work on reputability, so we can run the system automatically on auto-fetch VDI: in practical terms this (=improving reliability) means to follow the problems more rigorously TSz: and to investigate the problems as they happen; gradually it will reduce the problem VDI: this just means more/better TIO training TSz: and continuous training ======================================================================= 3. Status of the surveys (MRe, MHi) ======================================================================= ======================================================================= 4. Phase 3; are there any lessons we learn how to operate better VIRCAM@VISTA as a result from the Phase 3? (MAr) ======================================================================= MAr: we need to have the OB status for the scientific validation; so, we need access to the CASU reports TSz: do archival users already see the data? MAr: no, the query interface is still being developed TSz: some kind of a tutorial is needed; a public survey should be truly public and not secret MRe: they should show what kind of data are available MAr: one component of the data product is the "release description", showing the kind of data are in the package, and what is the sky coverage; part of our check is to verify the consistency between the description and the actual data set; the query interfaces automatically, when the data product is displayed, to show the meta-data, the address of the "release description", and the reference TSz: a problem is that the VIRCAM pointings are missed in the archive searches, unless the search radius is 1-1.5 deg VDI: this is a common problem for all instruments, especially those with large FoV, but also with small ones TSz: yes VDI: going back more closely to the original topic - some QC problems will be discovered during/after Phase III; how are we going to get them reported back to us? MAr: we don't have VISTA data quality queries yet, all problems that came up during Phase III were related to the submission process MRe: were there no problems at all with the data reported at stage of Phase III? MAr: no, we didn't have problems reported about the data quality; in addition - the La Silla - Paranal STC sub-panel member Nick Mellies said that almost all UltraVISTA data were reduced by Terapix successfully, with only 4% showing problems; we are waiting for a detailed report MRe: we should explicitly ask during the next review the PIs what were the data problems that they encountered ======================================================================= 5. Status of various software: SADT/OT/P2PP/etc (MRe, MHi, JEm) ======================================================================= TSz: SMi, is there any an input from T. Bierwith about the implementation of the push/pull factor in Orange? Once this comes we can start to discuss the priorities again; the night log tool will become available on Jul 26; Dario Dorigo will be up on Paranal from 26.7 to 02.08 to install the new night log tool on the offline machine MRe: ??????? waiting for the instrument package VDI: the new IP was just submitted TSz: Dario wanted also to look at this in July MHi: the new SADT version with 2MASS-based I-band magnitudes was just released; some tests were done and demonstrated that red stars should be avoided and the magnitude limit after that is the same as for the GSC but much 2MASS is much more homogeneous than the GSC MRe: will it be available for the next Phase 2? TSz: it should be tested at the telescope by SMi soon MHi: I can give you the new SADT, and prepare some OBs TSz: the rotated tiles must be tested; we should run VISTA for an hour with many short OBs generated with the new SADT, with DIT=1s SMi: just send me the OBs and I will run them JEm: I have slightly updated version for the VST, with only cosmetic changes; I need to check with Tomas before changing the Position Angle definition, though MHi: this will not change anything for the VISTA SADT, right? JEm: I can make a different version for each telescope, but I would rather it is the same for both VISTA and VST ??????? MHi: we are only talking about VISTA now JEm: I can split the it into two versions TSz: it is better, otherwise we mix two things MHi: OK, I will prepare some test OBs for Steffen MRe: remember that we still need to give Klara the new zero points for the VISTA ETC WH: they are prepared and were distributed last week. VDI: will do ======================================================================= 6. New/recent QC issues, data reduction/product issues (WHu, MIr) ======================================================================= WHu: new calibration database was sent to Paranal; a problem - the darks show larger RON for detectors 1-8; new plots for more NDITxDIT combinations on the QC page were implemented; another issue - Chip 13 is slowly changing; it looks like during the first 2 weeks the instrument was not entirely stable, may be not all parts had nominal temperature TSz: it is normal JLe: what was the change of the bias in Det 13? TSz: 40-50 adu JLe: the problem with detector 13 may be related to non-linearity WHu: we process ~10% of the science data to provide reference science products to be used in Phase 3. The processing of science data will be reviewed at the end of the year (and may not continue after that). SMi: what does it mean? WHu: It is possible that it will be decided that QC will not do any QC1 anymore. SMi: The deployment of the pull-push factors is now planned for around June 27; What is the fraction of the rejected OBs during QC1? MRe: after the automatic check of the ellipticity the fraction of the re-classified OBs is very low, the problem is being caught well at QC0 level on Paranal WHu: the most common problem for rejecting was ellipticity (but this is being checked on Paranal now) MRe: since Nov we had 1-2 occasions of rejected OBs per month due to QC1 checks; before the reclassification were much more common VDI: anything else we can improve in view of this change? MRe: with the implementation of the new night log tool it will show better any problems VDI: And the TIOs will have more time to check the data quality ====================== 5. Action Items review ====================== OLD AIs: -------- AI 2009-11: TSz should find out if the constraints sets could be carried with the OBs, to the OS and the fits header of the VIRCAM files. Otherwise it is not easy to propagate these data by interfaces. 2009-07-29: TSz iterating with R. Schmutzer, but DFI (T. Bierwirth) will also need to be involved. For surveys, given the small range of programs it is expected that there are not too many different OBs, and this might need to be followed in a different/manual way in the beginning this could imply a major change for the tools that cannot be done in the last moment. 2010-03-23: the keywords have been implemented except for the moon; there is a ticket with details; some testing done by VDI, no errors found; keep ongoing, try to complete during my next shift end of Mar early Apr. 2010-04-23: VDI reports one formula is missing 2010-05-27: Corresponding PPRS is still pending. VDI asks to keep this AI open until the PPRS is closed. 2010-06-29: No changes with respect to last meeting. VDI asks to keep this AI kept open/pending until the PPRS is closed. 2010-08-24: VDI not present at the meeting. It is not known if there was any progress. Therefore decided to keep this pending while waiting for final status from VDI. 2010-10-05: VDI: still pending, will force the issue with the SW during my next shift from Oct 19 to 30, 2010 STATUS: PENDING ** AI 2010-06: TSz+VDI to improve the TIO training, to prepare an operational manual 2010-05-27: Operations wiki has been updated. More training needed. 2010-06-29: Not much was done due to problems with IQ + intervention + work on IP. Keep the AI open. 2010-08-24: SMI comments that the training is an ongoing task. Therefore it seems that this could be closed. It is not clear about the status of the operational manual mentioned in the AI. Thus for the moment kept as PENDING. Comment from VDI and TSZ on the status required. 2010-10-05: to be kept open for now as a reminder STATUS: PENDING ** AI 2010-09: photometric zeropoints as calculated by the pipeline. 2010-05-27: Ongoing, see corresponding discussion this meeting 2010-06-29: TSZ points out that this is not well defined AI. MIR adds that the currently existing plots already would have shown the problems with zero points from pawprints to tiles. It is not clear whether this is about the zero points from Garching pipeline? WHU states that Garching pipeline has de-blending on. MIR: There is no significant increase in the scatter with respect to what one would expect from the random noise. It is agreed in the end to send one example data set and then the comparison should be done directly between Garching and CASU. 2010-08-24: This is still pending. WHU should send some example to JLE/MIR in order to do the checks and clarify what the problem actually is about. 2010-10-05: the example data were sent to MIr and JLe; the Cambridge pipeline results for the individual detector were as expected from "rms" noise considerations, though offset by 0.1 mag from the Garching results. JLe and WHu are attempting to identify the cause of this difference which is most likely due to differences in calibration frames used. JLe sent a new linearity curve to use; WHu reprocessed the data with the new linearity correction and there were no changes; JLe asked if he could get the calibration frames used from WHu. CASU use the "average" of all detector zero-points to monitor extinction/throughput. STATUS: PENDING ** AI 2011-01: SMi to contact Stefano Zampieri and to find out if the partial transfer of the VISTA data (only calibrations) can be implemented quickly, i.e. on a time-scale of four weeks? 2011-01-27: SMi sent an e-mail already to S.Z. 2011-06-08: done; it is running, the calibration data are transferred on nightly basis STATUS: CLOSED ** AI 2011-02: WHu to compare the ZPs from CASU and Garching from Nov-Dec 2011 to verify the scatter; TSz will acquire more standards during the technical nights in early Apr, after VISTA is back with the recoated mirrors; TSz will be up for the technical time. 2010-06-08 Duplication of AI 2010-09; still pending; CASU got some post-intervention data but the weather was too poor to get good ZP; leave this open until the next IOT in Aug 2011. STATUS: PENDING ** AI 2011-03: JEm to modify SADT by mid-Feb/end-Mar to include the I band predicted from 2MASS 2011-06-08: done STATUS: CLOSED ** AI 2011-04: TSz to derive/update the mag limits for the WFS stars in Apr, 2011, after the recoating 2011-06-08: not done yet but the observations in May showed that the 15 mag limit are just acceptable, so there is no need to update anything. Only for one UltraVISTA tile with 15.03 mag reference star was a problem but changing the contrast ratio fixed the problem; one good thing to do is to add which of the limits of the AO parameters can be redefined to improve the correction. STATUS: CLOSED -------- NEW AIs: -------- AI 2011-05: TSz to optimize the AO parameters to get better correction with faint stars; to be discussed with Will Sutherland. STATUS: NEW ** AI 2011-06: MAr et al. to prepare a memo to the management on the VISTA status and survey progress concerning the over-submission in the next period STATUS: NEW ** AI 2011-07: MHi to prepare test OBs with the new SADT; SMi to execute them STATUS: NEW ** AI 2011-08: WHu to send to Klara S. the new ZPs and sky values to update the ETC STATUS: NEW ** AI 2011-09: TSz to investigate if the control loop parameters for the azimuth axis can be modified to speed up the telescope preset. STATUS: NEW ** AI 2011-10: VDI and MRe to investigate if the comments written by TIOs can/should/mast be conveyed to the PIs, especially comment for aborted OBs. This was prompted by user's question. 2011-06-09: the issue was discussed; the comments are relevant only if the aborted OB generated files, if not thee is no need to convey anything but if there were files the comments may help to decide if the data are of any use; decision on this should be postponed until the new night report tool is installed in July; this AI to be kept as a reminder. STATUS: NEW (really, ON HOLD) ** AI 2011-11: JLe to check if the data of the last OB status change is written in the file headers (so the users can see/understand if it is current) STATUS: NEW ======================================================================= 10. Date of the next meeting, proposed in the interval Aug 20-30, 2011 ======================================================================= To be discussed when it grows nearer.