----------------------------------------------------------------------------- Mauna Loa Solar Observatory Observer's Log ----------------------------------------------------------------------------- Tue Apr 26 16:26:35 GMT 2022 Year: 22 Doy: 116 Observer: mcotter WEATHER COMMENT: mcotter: Tue Apr 26 16:31:09 GMT 2022 Temp: 46.2f, Humidity: 10%, Pressure: 28.638in, Wind: 8mph from 161degs, Skies: Heavy rain and overcast skies in the Saddle Valley up to approximately 8000' elevation level. Somewhat clear skies above 8500' elevation level, though the sky is a bit hazy and there are light sporadic bands of high altitude Cirrus clouds scattered across the sky in various areas. Pinkish gray inversion layer visible on the horizon just above Haleakala. ___end___ GENERAL COMMENT BY mcotter: Tue Apr 26 16:43:47 GMT 2022 Opened windows upstairs ___end___ GENERAL COMMENT BY mcotter: Tue Apr 26 16:43:57 GMT 2022 PM Blew off Kcor O1 ___end___ GENERAL COMMENT BY mcotter: Tue Apr 26 16:46:29 GMT 2022 PM Blew off UCoMP O1#2. Removed 01#2 from instrument and installed 01#1. Blew off Ucomp 01#1. Ucomp 01#2 stowed in holding fixture and covered with a lens cloth in the clean room. ___end___ Tue Apr 26 17:00:54 GMT 2022 Kcor Focus/alignment program exited WEATHER COMMENT: mcotter: Tue Apr 26 17:02:26 GMT 2022 High altitude Cirrus cloud bands persist on the eastern horizon making the sky too bright to observe at this time. ___end___ Tue Apr 26 17:23:08 GMT 2022 Kcor Focus/alignment program exited Tue Apr 26 17:25:18 GMT 2022 Running UCOMP Cookbook all_wavelength_coronal_flat.cbk line 0 GENERAL COMMENT BY mcotter: Tue Apr 26 17:30:09 GMT 2022 Though the sky is still a bit bright both Kcor and Ucomp are now observing. ___end___ Tue Apr 26 17:44:50 GMT 2022 Running UCOMP Cookbook all_wavelength_coronal.cbk line 0 Tue Apr 26 17:51:32 GMT 2022 UCoMP Paused for clouds Tue Apr 26 18:22:10 GMT 2022 UCoMP Restarted from pause Tue Apr 26 18:22:42 GMT 2022 UCoMP Paused for clouds Tue Apr 26 18:24:01 GMT 2022 UCoMP Restarted from pause Tue Apr 26 18:31:20 GMT 2022 Running UCOMP Cookbook all_wavelength_coronal.cbk line 6 GENERAL COMMENT BY mcotter: Tue Apr 26 19:03:10 GMT 2022 High altitude Cirrus clouds have moved into the viewing area. Kcor has been stopped. ___end___ Tue Apr 26 19:09:04 GMT 2022 Running UCOMP Cookbook all_wavelength_coronal_flat.cbk line 0 Tue Apr 26 19:14:02 GMT 2022 Running UCOMP Cookbook all_wavelength_coronal_flat.cbk line 2 Tue Apr 26 19:14:03 GMT 2022 UCoMP Paused for clouds Tue Apr 26 19:31:13 GMT 2022 UCoMP Restarted from pause Tue Apr 26 19:33:11 GMT 2022 Running UCOMP Cookbook all_wavelength_coronal_flat.cbk line 3 Tue Apr 26 19:38:15 GMT 2022 UCoMP Paused for clouds Tue Apr 26 19:40:02 GMT 2022 UCoMP Restarted from pause Tue Apr 26 19:51:14 GMT 2022 Running UCOMP Cookbook all_wavelength_coronal_flat.cbk line 0 Tue Apr 26 19:55:14 GMT 2022 Running UCOMP Cookbook all_wavelength_coronal.cbk line 0 **UCoMP PROBLEM COMMENT BY mcotter** : Tue Apr 26 21:10:31 GMT 2022 An image came up in the Ucomp--Controller GUI that I had not observed before. The image had the appearance that one of the optics, or one of the filters or the occulter had not fully followed thru with its travel and was badly misaligned. I immediately put the instrument in the "Clouds" position. Because the last item worked on in this beam was the 01 optic I checked the 01 position via the Mech-Controller GUI. The 01 appeared to be in the correct position. I went to the dome and physically checked that the 01 was secure and in the correct position and it was. I called Ben and texted him photos of the GUI images and told him that I suspect that one of the filters might not have been in the correct state. Using the Mech-Controller GUI, Ben instructed me in how to verify the filter positions. I completed this task and the filters wheel appeared to be in the correct position. The anomaly continued so the only thing we could think of is that the occulter may not have been in position. I engaged the "Center Occulter" button and when it came up the GUI image was completely black. The shutter was in the beam for an unknown reason. When I removed the shutter from the beam the image was completely over saturated. I raised the camera gains from 800 to 8000. The image was now black so I slowly adjusted the settings incrementally down to 4000 and was able to get an image showing the occulter very far out of position. I forgot to note the occulter position, but Ben said he could check that figure later. I was able to adjust the occulter back to center and exited the GUI screen. When the Ucomp-Controller screen came back up after a few minutes it again showed the occulter out of position toward the bottom. I again engaged the Center Occulter button and when the GUI came up the controls to move the occulter were unresponsive. At this point Ben told me to shut everything down and restart the computer. Before restarting the computer I went to the back of the racks and physically disconnected the power labeled "Standa", waited 30 seconds and reinserted the plug to re-energize, as instructed by Ben. I have restarted the Ucomp instrument and it appeared to come up normally. I engaged the Center Occulter button and when the GUI came up the shutter was again in the beam. I removed the shutter from the beam and immediately the GUI image was over saturated. I quickly put the shutter, cover and diffuser into the beam. The GUI image was then black. I removed first the shutter and when the image came up it was dark gray. The position of the occulter was 59.8 x 3.97. When I had the occulter centered previously I remembered the location was "60.4 x 3.27. I returned the position of the occulter to these values. I then first removed the cover from the beam and then the diffuser from the beam. The occulter appeared near centered. During this time clouds entered the viewing area so I paused the Ucomp and Kcor instrument unitl conditions improved. When I restarted Ucomp I again checked the occulter position and adjusted it to 60 x 3.27. The occulter now appeared well centered. I exited the Center Occulter GUI and disengaged the Clouds pause button. The instrument appears to be running normally, but the Mech-Controller indicated that the shutter is in the beam. When I tried to manually remove the shutter from the beam it immediately returns to "Shutter In Beam", as if it is latched into place. I stopped the recipe early and stopped the program. Ben called me back while I was working on this problem. He asked me to restart the Ucomp-Controller program and engage the Center Occulter button. When the center occulter GUI came up Ben asked me to record the position of the Mech-Controller Mech Pos field values. From top to bottom they read 135, 45, 0, 60, 3. Ben then instructed me to exit the center occulter GUI and verify if the values changed. The only value to change was the occulter position which went to -3. Ben instructed me to try to engage both the Shutter in and Shutter out buttons to see if they worked. They did not appear to have any effect. Ben asked me to verify if there was another Mech-Controller program running, and indeed there was a second one opened but folded down on the bottom of the screen. At this point Ben instructed me to close all Mech-Controller programs and the Ucomp-Controller program. I restarted the Ucomp-Controller program. When the program came up ben asked me to engage the center occulter button and open the GUI. When the center occulter GUI opened the I centered the occulter and exited the GIU. When the controller GUI opened I started a receipe. Ben asked if the "Shutter In" was still engaged and it was. Though the shutter was in beam an image of the occulter could still be observed. Ben instructed me to go to the dome and move the spar off sky, then put the Diffuser and the Cover into the beam. I ensured that the diffuser and cover were in place then left the Ucomp instrument in an OFF state. Ben was in transit and told me he would call back when he got home. ___end___ GENERAL COMMENT BY mcotter: Tue Apr 26 21:12:23 GMT 2022 Though the sky is a bit bright, Kcor is again on sky and observing. Ucomp remains off until Ben returns. ___end___ Tue Apr 26 22:02:34 GMT 2022 Running UCOMP Cookbook all_wavelength_coronal.cbk line 0 Tue Apr 26 22:17:54 GMT 2022 Running UCOMP Cookbook all_wavelength_coronal.cbk line 11 Tue Apr 26 22:21:41 GMT 2022 UCoMP Paused for clouds Tue Apr 26 22:46:51 GMT 2022 UCoMP Restarted from pause GENERAL COMMENT BY mcotter: Tue Apr 26 22:50:45 GMT 2022 The mouse stopped working (second time) so I restarted the KVM computer. ___end___ Tue Apr 26 22:56:01 GMT 2022 Running UCOMP Cookbook all_wavelength_coronal.cbk line 0 Tue Apr 26 23:03:43 GMT 2022 UCoMP Paused for clouds GENERAL COMMENT BY mcotter: Wed Apr 27 00:01:08 GMT 2022 I observed the dome while tracking the spar to see if and when the dome auto rotation ceases to work. At approximately 12:40 HST the dome made a last rotation via the automated tracking. The dome advanced approximately 45 deg so that the dome slit was facing toward the south. After this the spar continued to advance to the point that the two outside position sensors became overshadowed by the dome and finally the Ucomp instrument started into the shadow of the dome. The Kcor instrument was still on sky but the Ucomp instrument was being blocked by the dome. I manually rotated the dome so that the dome slit was facing west and left the the dome controls in manual. I took several photos of the spar/instruments position before manually advancing the dome position so that we may be able to establish how the sensors should be set up. For now we should know that the dome auto alignment is only reliable to approximately 12:45 HST, at which point the dome should be manually rotated into a westerly position. ___end___ UCoMP COMMENT BY mlso: Wed Apr 27 00:35:22 GMT 2022 It is not exactly clear what happened this morning with the occulter not properly moving into the beam.. But it looks like 2 copies of the mechanism-controller were started and commands for mechaims controls got mixed up between them. Typically if the ucomp main gui/sequencer need to make a mechaims move it checks if the mechanism-controller is running and spawns it if it isn’t. Normally this seems pretty robust, as this check happens multiple times a day. But this time we ended up wiht a second copy and one of the copies seems to have loaded a bad configuraiton particularly related to where the occutler shoudl be driven to place it in the beam. With the two mechanism-controllers running it looks like the one with the bad state ended up processing some or all of the requests from the main gui. While I think the observer was seeing status on the other controller. This lead to some synchroization errors/confusion behavior as it wasn’t obvious to peices of code were running. Around this time we had a couple occulter move requests that look longer than expected (shoudl be able to run the full range in ~30 seconds). It is not clear why these requests were slow; but I suspect this was related to sending request to the other controller. 4/26/2022 7:47:36 AM: Waiting over 1 minutes. for Occ X to In _ 0 4/26/2022 8:26:06 AM: Waiting over 1 minutes. for Occ Y to Get Pos _ 0 4/26/2022 8:28:54 AM: Waiting over 1 minutes. for Occ Y to Move Rel _ 0 I think the ultimate solution to fixing this problem was to close the extra mechamis controller and make sure it loaded with a good config. Going forward I think we should try to to make sure there is only one copy of the mechanism-controller running. And stopping/restarting all of the labview code if it isn’t clear which one to close. ___end___ Wed Apr 27 00:41:47 GMT 2022 UCoMP Restarted from pause Wed Apr 27 01:01:27 GMT 2022 Running UCOMP Cookbook all_wavelength_coronal.cbk line 0 Wed Apr 27 01:12:18 GMT 2022 UCoMP Paused for clouds GENERAL COMMENT BY mcotter: Wed Apr 27 01:26:21 GMT 2022 The has gotten quite bright and both instruments remain paused. ___end___ GENERAL COMMENT BY mcotter: Wed Apr 27 02:09:06 GMT 2022 The sky remains fairly bright. Both instrument remain paused at this time. ___end___ UCoMP COMMENT BY mcotter: Wed Apr 27 02:55:57 GMT 2022 Provided measurements of Ucomp 01 holding mechanism to determine how it physically engages with the 01#2 lens frame. ___end___ Wed Apr 27 03:00:14 GMT 2022 UCoMP Restarted from pause GENERAL COMMENT BY mcotter: Wed Apr 27 03:10:15 GMT 2022 The sky was a bit bright all day with high altitude Cirrus clouds. Instruments needed to be paused several times during the course of the day for passing Cirrus clouds bands. Some problems with Ucomp in the morning. Some observing was done, but sporadically. ___end___ ONSITE STAFF: