----------------------------------------------------------------------------- Mauna Loa Solar Observatory Observer's Log ----------------------------------------------------------------------------- Fri Nov 15 16:32:08 GMT 2019 Year: 19 Doy: 319 Observer: waters WEATHER COMMENT: Observer: Fri Nov 15 16:44:09 GMT 2019 Temp 41F with windspeed at about 7-13mph from the South. Sky looks clear. ___end___ GENERAL COMMENT BY waters: Fri Nov 15 16:58:26 GMT 2019 PM Blew off Kcor O1 ___end___ GENERAL COMMENT BY waters: Fri Nov 15 16:58:37 GMT 2019 Windows opened upstairs ___end___ GENERAL COMMENT BY waters: Fri Nov 15 17:32:29 GMT 2019 Kcor above dome ___end___ Fri Nov 15 17:33:26 GMT 2019 SGS Alignment complete Fri Nov 15 17:34:31 GMT 2019 Kcor Focus/alignment program exited Fri Nov 15 17:35:05 GMT 2019 KCOR Start Synoptic Patrol Fri Nov 15 17:49:09 GMT 2019 KCOR End Patrol GENERAL COMMENT BY waters: Fri Nov 15 17:51:50 GMT 2019 Seeing a slight oscillation in the spar with the RA bouncing between 10 and -10 arcsec and then calming down and then starting again. ___end___ GENERAL COMMENT BY waters: Fri Nov 15 17:52:30 GMT 2019 Lost the RA, manually went to move it back on the Sun ___end___ Fri Nov 15 17:53:04 GMT 2019 KCOR Start Synoptic Patrol Fri Nov 15 17:56:12 GMT 2019 KCOR End Patrol GENERAL COMMENT BY waters: Fri Nov 15 18:14:52 GMT 2019 Lost the RA again, Bens' going to do a little inspection real quick of the balance system. ___end___ GENERAL COMMENT BY waters: Fri Nov 15 18:22:37 GMT 2019 Tightened the RA clutch by a quarter turn. Currently the guider seems to not be oscillating as much now that we've tightened it. The Ra is a lot harder to move now, which is at the limit of what I can move. ___end___ Fri Nov 15 18:28:43 GMT 2019 Kcor Focus/alignment program exited KCOR COMMENT BY waters: Fri Nov 15 18:34:16 GMT 2019 KCor had a SPii error after we were done with the spar. Currently rebooting the whole system. ___end___ Fri Nov 15 18:43:18 GMT 2019 KCOR Start Synoptic Patrol Fri Nov 15 18:43:40 GMT 2019 SGS Alignment complete Fri Nov 15 18:49:43 GMT 2019 KCOR End Patrol Fri Nov 15 18:50:20 GMT 2019 SGS Alignment complete Fri Nov 15 18:49:58 GMT 2019 KCOR Start Synoptic Patrol Fri Nov 15 18:59:09 GMT 2019 KCOR End Patrol Fri Nov 15 18:59:48 GMT 2019 SGS Alignment complete Fri Nov 15 18:59:25 GMT 2019 KCOR Start Synoptic Patrol Fri Nov 15 19:29:22 GMT 2019 KCOR End Patrol GENERAL COMMENT BY waters: Fri Nov 15 19:32:38 GMT 2019 Lost the RA twice; the second time was right after I got it back on the Sun. ___end___ Fri Nov 15 19:33:32 GMT 2019 KCOR Start Synoptic Patrol Fri Nov 15 19:57:53 GMT 2019 KCOR End Patrol GENERAL COMMENT BY waters: Fri Nov 15 20:11:57 GMT 2019 Testing on the integral time setting on the RA PID settings seems to give better results; currently running it at 0.020 instead of 0.014 ___end___ Fri Nov 15 20:12:59 GMT 2019 SGS Alignment complete Fri Nov 15 20:12:39 GMT 2019 Kcor Focus/alignment program exited Fri Nov 15 20:13:25 GMT 2019 KCOR Start Synoptic Patrol Fri Nov 15 20:31:59 GMT 2019 KCOR End Patrol Fri Nov 15 20:54:26 GMT 2019 KCOR Start Synoptic Patrol Fri Nov 15 20:56:00 GMT 2019 KCOR End Patrol GENERAL COMMENT BY waters: Fri Nov 15 21:10:54 GMT 2019 Lost the RA again, went back upstairs. Noticed that when we hit the slew button on SGS, it wouldn't respond and when it did, it took many times of hitting the button to get it to work. Removed five pounds from the spar balance arm, at vertical it seems to be fairly balanced. However, we just lost it again. Currently have lost the RA twice in typing this. Restarting the SGS computer to see if it might help. ___end___ GENERAL COMMENT BY waters: Fri Nov 15 21:16:51 GMT 2019 SGS froze on restart, Ben pulled the plug on the Nema box upstairs. ___end___ GENERAL COMMENT BY waters: Sat Nov 16 00:27:10 GMT 2019 Looks like some of the clouds coming in are a little low, closing the dome just in case. ___end___ GENERAL OBSERVATORY COMMENT BY berkey: Sat Nov 16 02:11:40 GMT 2019 Lots of messing with the guider/RA train today with the hope to address tracing issues that caused the spar to loose tracking. It appears the root issue was related to the slipping of the RA clutch. Such that the guider would attempt to drive the spar West (RA+), push to hard against the clutch and cause it to slip until the RA unloaded the clutch, would catch again and tracking could be restored. Unfortunately this particular behavior lead to some misleading observations. The GUI feed back suggested we were pushing too far west. The known slight spar miss-balance this morning with the friction roller released (or any other condition where gravity won) pulled the spar West as well. Yet when we went upstairs to realign the spar it tuned out to be East of the sun. Pushing the spar back into position would tack the load off the clutch and allow motion to be restored!!!! It all just seemed like a conflicting set of facts. If the observer stayed down stairs, motion through the clutch could be recovered by a -10V dec slew or a stop motors standby command. However based on the apparent information that the spar needed a +RA kick these seemed like confusing inputs. However, this all made sense with a visual inspection of the RA clutch. What was actually happening the whole time was resistance in the drive train started to overwhelm the clutch, at which point the sun would start to naturally drift west of the guider. Sensing this the guider would increase the voltage (speed) of the RA motor and reinforce the slipping of the clutch. The guider would see this as a bigger RA error and send the voltage to max which we assumed would stay railed until the sun drifted completely out of the guiders filed of view, at which point with no sun to track the guider would go into open loop and either follow at the solar rate or if the clutch was still slip just stay parked. Before and after finding the actually underlying clutch slip issue, many things where attempted to restore functionality and allow for observations. During this we also noticed that there was rattling/ticking noise coming from the work gear assembly. Without removing the RA drive assembly from the pier, we were unable to locate the exact source of the noise, but it seems like it may be either slippage in the 8104-SW-0314 assembly or one of the bearings within that assembly. It seems like this is leading to some jerkyness in the motion of the complete RA assembly but it does not seem to be directly related to the issue we were seeing in which the RA was loosing tracing. Test Checks and tweaks today in order: -Attempt to characterize how far out of balance we were with too much weight on the end of the SBS system and too much at the base of the spar. Found about 405 inch pounds out of balance with the spar vertical and about 150 inch pounds when horizontal/ pointing east. -Tighten the friction roller to prevent the spar form dropping or sagging. In hindsight this may have been a bad thing to do since it now meant that any slipping in the RA train would happen in the degraded clutch and not on the friction roller. -Removed 5 pounds off the SBS to bring the spar when vertical in to just about perfect (no preload) balance. -Slightly adjusted the RA integral gain up from .014 to .02 this seemed to have a lot of benefits where the spar recovered from small deviations quickly and the RA error signal (guider telescope output) was much smoother and looked more like the Dec signal. After this test the guider software was restarted which automatically reset it to .014 This let us run for a couple hours without issue. Seemed like time for RA champagion Then around 11am we started getting RA issues again that persisted for a few hours. The guider would track happy for up to about 5 minutes and then the clutch would slip and we would be stuck in the above scenario. -Tension on the fiction roller was reduce to more or less match what we had this morning before any intervention. -In depth inspection of the RA train took place to the point were we found the clutch slippage. -Checks were made to see if the meshing of the worm gear to the drive gear were leading to the slippage by manually increasing/reducing the spring force holding them together. No change in behavior. -Eventfully it was found that the fiction roller was slightly misaligned/binding with some of the structures in the RA drive train. To address this, we released the tension on the friction roller handle and carefully reapplied tension while manually guiding the roller back into alignment. By this time we started to get passing clouds, but the guider was able to track the sun continuously for about 1 hour 1:30-2:30 with no RA hicciups. So it appears that aligning the fiction roller solved the issue and got the guiding back. But we may also had just waited late enough in the day that some sort of underlying balance issue got us past the RA clutch hiccups and it just coincided with the roller alignment. Before heading out we planned to get a video of the rattle/chatter with the dome but found it seemed to be a produce of guiding with lots of velocity/direction changes; but didnt occur under constant slew speeds. We did find that the clutch slipped when vertical :(. To address this we started moving weights around the spar and eventual removed 5 pounds from the back end and added the 5 pounds back on to the SBS. This left use enough out of balance that the friction roller needed to be re-tightened a bit. We now find that we can slew the spar in RA (+/- 10) from Sun up until about ~3-4pm before we see the RA clutch slip. When this happened we could regain slewing by adding the small led weight toward the found of the spar. At this point we do not have a safe place to hang that weight, so if this occurred in the future a more stable mounting point should installed. TODO going forward: We need to figure out the rattle/ticking from the worm. PM or otherwise inspect the clutch to get it to not slip so much. This may be an adjustment of the clutch itself or it may be we have some misalignments ala friction rollers elsewhere in the RA TRain. ___end___ KCOR COMMENT BY waters: Sat Nov 16 02:19:48 GMT 2019 Kcor hepa was off for most of the afternoon to reduce the noise load while we were trying to understand the source of guider noises. They are almost impossible to hear with the RA covers on and the hepas make it difficult even with the coves off. ___end___ ONSITE STAFF: berkey, waters