Instrument Log    OCO

OCO   C1       Lamont, OK

Date

Technician

Comments

7/1/16

KT

The window had dry water spots and dead bugs on it and needed to be cleaned. At 1638 GMT placed the instrument into HOLD, and removed the window. Cleaned the window and installed at 1648 GMT. At 1650 GMT performed SW Status Reinitialize. System is online and ready to begin normal operations. RTS at 1658 GMT.

6/16/16

KT

Instrument mentor requested that the OCO disk needs to be changed out. At 1925 GMT after instrument mentor unmounted the OCO disk, the disks were changed out. At 1931 GMT notified instrument mentor that the disks were changed out and the new disk was ready to be mounted. RTS at 1940 GMT.

6/16/16

KT

Instrument mentor stated the OCO had quit working. At 1820 GMT after the instrument mentor vented the system, it was found that the carriage assembly was stuck next to the motor unit again, having gone past the relay. Rotated the motor spindal and moved the carriage housing West away from the motor assembly. Mentor then began to start up the instrument again. At 1907 GMT the instrument was online and scanning. RTS at 1910 GMT.

6/13/16

KT

Instrument mentor stated that the instrument ran and collected data up until 6/12/16 at 2000 GMT. At 1700 GMT mentor vented the instrument. Removed the two covers and cleaned the rails that the carriage slides on and rotated the spool to move the carriage 7 inches west of the motor and power cycled the instrument. After the power cycle was completed, mentor restarted the instrument and the two panels were installed. At 1721 GMT mentor started the pump to build pressure back up into the instrument. After pressure in the instrument was normal, mentor started the program. System came online and was operational. RTS at 1939 GMT.

6/10/16

KT

Instrument mentor stated that the IFS in the OCO container had stopped scanning. At 1400 GMT inspected the encoder LED's and all were green. At 1644 GMT Chris Martin assisted Jean-Francois and power cycled the instrument. There was no change in status. At 1845 GMT while working with Jean-Francois I removed the East cover and noticed that the IFS125 Scanner was stuck near the motor. Rotated the spindle and moved the Scanner a few inches from the motor and pressed the IFS125 reset button. The scanner moved to it's normal position. Jean-Francois started the pump and instrument. the instrument took a successful scan at 1938 GMT. System is online and operational. RTS at 1940 GMT.

5/24/2016

MS

Hatch has not been closing properly.  Upon mentors request, at 1440 GMT, checked the copper loops and cleaned them, checked the foam inserts.  RTS at 1455 GMT.

5/16/16

KT

Rain began falling and the hatch was still open. Observed the computer display and it stated that the hatch was closed. At 1445 GMT typed in the command for the hatch to close. At 1448 GMT the hatch was closed. Called instrument mentor to let her know what actions were performed. RTS at 1500 GMT.

5/3/2016

MS

Instrument was not getting enough light.  Upon mentors request, the pass through window needed to be cleaned.  Mentor closed hatch and parked the instrument at 1545 GMT.  Took out pass through window and cleaned, reinstalled window and contacted mentor so that she could take the instrument out of home position and track the sun.  RTS 1600 GMT.

4/15/2016

MS

Noticed that the wind speed indicator was sticking.  Replaced bad bearing on the wind speed indicator,  RTS at 2051 GMT.

3/14/2016

MS

During PM rounds noticed that the Hatch was closed and the Tracker was in home position.  Contacted mentor at 1455 GMT. At 1512 GMT the Hatch opened and the Tracker located the sun.  RTS at 1512 GMT.

3/1/2016

MS

Noticed during PM rounds at 1606 GMT that the hatch was not opened all the way, mentor was contacted.  Mentor started working on the instruments hatch issue at 1745 GMT.  RTS at 1800 GMT.

2/29/16

KT

The laser cradle appeared to be hung up which caused the instrument operations to quit. At 1645 GMT removed the cover plates and inspected the laser cradle and pulley cables. There was a slight fray on the pulley cable which the mentor was informed of and at this time it did not seem to be an issue. The laser cradle had moved passed the stop point and needed to be moved forward slightly. Used rubbing alcohol and kimwipes and cleaned grime from the guide rails. Installed the cover plates and instrument mentor began start up mode. At 1715 GMT the instrument was online and began normal operations.

2/26/2016

MS

Notice that the hatch was closed and the dome in home position at 1555 GMT, 2/26/2016.  The Sensor Vacuum reading was very low.  Contacted Mentor about the dome being in home position and closed.  The Mentor noticed that the Sensor Vacuum reading was very low, therefore the Mentor has fixed the issue.  RTS 1750 GMT, 2/26/2016.

1/29/16

KT

Instrument Mentor was experiencing issues of getting the OCO hatch to close. At 2228 GMT after mentor had placed the instrument into hold, I used the hand held and was able to rotate the dome but could not get any response from the hatch. During inspections, there was no apparent damage to the hatch, gears, contact points. Removed the hatch and adjusted it to the close position and connected hatch to the dome. 2250 GMT used the handheld and was able to open and close the hatch. Contacted instrument mentor and she started the the program. All appeared operational at this time. RTS at 2300 GMT.

12/15/15

MS

Computer needed to be restarted following scheduled software and security patch updates.  At 1404 GMT computers were updated and rebooted by the IT department, some data might be lost.

10/19/2015

MS

Dome was out of position at 1442 GMT.  Put dome in home position and put in the correct commands to get the tracker back in its proper alignment.  RTS at 1457 GMT.

9/4/15

KT

Instrument mentor stated that there was an internal error with the instrument and requested assistance to troubleshoot. At 1523 GMT instrument mentor requested that I remove the foam panels and remove 2 panels to allow access to the inside of the instrument. The scanner was locked up and would slide back and forth on the rails due to build up on the rails. Used alcohol and cleaned the build up off the rails, checked the laser to observe it was outputting. Mentor then started the instrument and the laser scanner moved back and forth on the rails as it should. Mentor restarted the system as I placed the panels back into position and reinstalled the foam panels. At this time instrument is online and operational. RTS at 1633 GMT.

9/4/15

KT

Instrument mentor sent a new Zeno battery out that needed to be installed. At 1245 GMT stopped the program and changed out the Zeno battery. There was an issue with the new battery that was ordered. The battery terminals on the new battery were larger than the terminals on the spare and original battery. Changed out the terminal connectors and connected to the new battery. The battery voltage was measured at 13.6 VDC. RTS at 1300 GMT.

9/1/15

KT

Instrument mentor noticed that the Zeno battery voltage was reading between 10.6 VDC and 11.2 VDC and was dropping. Instrument mentor requested that the battery be changed out. At 1509 GMT performed a SW Shutdown quickly and was able to change out the battery. While system was down the window was removed and cleaned as well. At 1520 GMT installed the window and restarted the program. Called instrument mentor to she if all the values looked good. System was returned to service and is operational at this time. RTS at 1539 GMT.

7/9/15

KT

Instrument mentor notified that the shutter was not open and she could not manually open it. At 1835 GMT instrument mentor stopped the program. Mentor suggested that the copper loops be checked. Unable to open the shutter with the hand held controller so I removed the shutter and was then able to use the hand held controller to rotate the dome to 270 degrees to reach and inspect the copper loops. Both copper loops were intact but I did notice corrosion on the contact points of the copper loops so I used sandpaper and cleaned the copper loops. Placed the dome back into the home position and then reinstalled the shutter. At 1900 GMT used the hand held controller and was able to open and close the shutter. At 1905 GMT started the program and contacted instrument mentor and informed her that the shutter was operational and that the system was online. RTS at 1915 GMT.

6/10/2015

MS

Due to a scheduled power outage the OCO program was shut down at 1314 GMT.  Restarted OCO program at 1614 GMT.  RTS at 1629 GMT.

6/4/15

KT

After the OCO was powered on from the power outage there was an issue on getting the laser stable. Debra Wunch talked with Jean-Francois and it was recommended that the laser be powered down and allow to cool for 30 minutes and then to power back on. The laser was powered off at 2030 GMT. At 2100 GMT the laser was powered back on and took roughly 10 to 15 minutes then became stable. System is online and operational. RTS at 2130 GMT.

6/4/2015

MS

Restarted OCO programs at 1900 GMT.  RTS at 1915 GMT.

6/2/2015

MS

Dome was not in line with the sun.  At 1538 GMT realigned dome using instructions.  RTS 1553 GMT 6/2/2015.

5/14/15

PD/KT

Notified by Mentor OCO camera on roof of enclosure directed at OCO dome is only operable part of the time.  Mentor request camera R/R with one previously sent to SGP.  Computer dept. help set camera IP address and configuration settings provided by Mentor.  Shut down camera at 1540 GMT.  Removed fish eye lens from old camera, cleaned and installed on new.  Installed new camera and put spiral protective covering on section of camera power/comm cable exposed to outside elements.  Connected laptop to network switch and was able to view camera image and adjust as needed.  Notified Mentor.  RTS at 1640 GMT.  Old Startdot Industries, camera ID 0030F4C10A5A.  New: Startdot Industries model CAM-SL640, camera ID 0030F4C801EA.  Old camera is stored on shelf in OCO enclosure. 

3/10/15

PD/KT

Notified by Mentor hatch did not close last night.  Hatch motor input wires only measures ~1.3VDC when hatch is prompted to open or close.  Troubleshooting found poor contact between one of the copper loops and the base it mounts to.  Retightened copper loop to base.  Voltage at drive motor input wires now measure 10VDC to open and 8VDC to close hatch. 

3/9/15

PD/KT

Mentor reports OCO dome acting up over the weekend.  Hatch is not opening or closing correctly.  Mentor requests copper loops inspected.  Nothing obvious is wrong with copper loops, etc.  Cleaned with sandpaper and reshaped loops.  Hatch will move correctly using hand held controller.  RTS at 1552 GMT.

2/25/15

KT

The dome was found facing the wrong direction opposite the sun. At 2115 GMT during inspection, one of the copper loops was found broken. Placed the software in hold, removed and replaced the copper loop and reinitialized the program. System is online and operational. RTS at 2135 GMT.

2/11/15

KT

Instrument mentor requested that the disk be removed and replaced. At 1653 GMT removed and replaced the storage disk. RTS at 1724 GMT.

2/9/15

KT

The hatch was not fully opened. At 2130 GMT stopped the program and placed the dome into home position. Opened and closed the hatch without issues. At 2137 GMT reinitialized the program and brought the instrument online. RTS at 2145 GMT.

2/3/15

KT

The hatch was opened only half way. At 1553 GMT placed the dome into the home position, then placed the software on hold. Used the handheld and inspected the hatch. No issues could be found at this time. Rotated the dome back to the home position then reinitialized the program. System is online and operational. RTS at 1608 GMT.

1/30/15

KT

Instrument mentor could not get the dome to rotate. At 1525 GMT rotated the dome and found the foam pieces  that fit inside the copper loops on the ground. Reinstalled the foam. Added tape around the base of the foam to keep from falling out. Notified mentor of corrective actions taken. Mentor then placed the instrument into RUN mode. System is online and operational. RTS at 1540 GMT.

1/20/15

KT

The hatch was not opened all the way. During inspection there was a piece of foam that fell out of the copper loop. Replaced the foam in the copper loop. At 1610 GMT restarted the program. At 1615 GMT instrument is online and operational. RTS at 1620 GMT.

1/15/15

KT

The instrument was running but the hatch was not fully opened. At 1650 GMT placed the instrument into hold and inspected the hatch and hatch assembly. Could not find anything out of place. Inspected the tracks and the slides for the hatch, all appeared intact. At 1700 GMT used the handheld and opened and closed the hatch several times. Was able to open the hatch fully. At 1710 GMT reinitialized the program. Instrument was online and began normal operations. RTS at 1715 GMT.

12/29/14

KT

The OCO external mirrors and internal window needed to be cleaned. After receiving instrument mentors approval the instrument was placed into holding at 1917 GMT for internal and external cleaning. At 1945 GMT after the cleaning was completed the instrument was placed back into service. RTS at 1945 GMT.

9/24/14

KT

The program was stopped for a planned power outage. At 1410 GMT stopped the program. At 1700 GMT the program was started. System is online and operational. RTS at 1705 GMT.

9/10/14

KT

Instrument mentor requested that the OCO dome and hatch be inspected due to issues upon opening and closing. During inspection one of the copper loops had broken and needed to be replaced. Removed and replaced the copper loop. Reinitialized the program. System is online and operational. RTS at 1640 GMT.

8/26/14

CAM

Instrument mentor requested disk changed out as part of OCO-2 flyover.  Mentor stopped program and unmounted disk at 1945 GMT.  Replaced disk at 1950 GMT and mentor restarted software.  Delivered disk to S/R for overnight delivery.

8/20/14

KT

The window had dust, spider webs and a dead beetle which was located in the center of the window. At 1410 GMT moved the dome to the home position and closed the hatch and placed the program into hold. Removed and cleaned the window and placed the window back into position. At 1420 GMT performed system reinitialize. At 1425 GMT system was online and performing normal operations.

7/25/14

KT

The dome was pointing North East. After the window was cleaned the instrument program was reinitialized. At 1403 the program started up and the dome moved to its correct position. System is online and operational. RTS at 1418 GMT.

7/25/14

KT

The window was dirty and needed to be cleaned. At 1348 placed instrument into hold. Removed and cleaned the window and placed back into position. RTS at 1403 GMT.

3/31/14

PAD

OCO display shows dome position is 107.7 deg but actual is ~ 20 deg.  Put system in hold mode then reinitialized.  Dome moved to correct position.  RTS at 1516 GMT.

3/18/14

KT

The window needed to be removed and cleaned. At 1528 GMT placed the instrument into hold and removed the window for cleaning. After installing the window the instrument program was reinitialized. System is online and operational at this time, RTS at 1543 GMT.

2/25/14

KT

Instrument mentor requested that the OCO disk be changed out. At 1500 GMT removed and replaced the OCO disk. Took removed disk down to shipping/receiving to be sent back to mentor. RTS at 1515 GMT.

12/23/13

CAM

Instrument mentor sent email advising that OCO dome had stayed open during the weekend ice storm.  She had tried to close it remotely but it would not close.  Found that dome was open and covered in a heavy layer of ice.   However, the suntracker appeared to be in good condition with no ice buildup.  Chipped all ice off the dome and could then move dome in azimuth direction, but the hatch still would not close.  Found one of the copper loops was not securely mounted to its bracket and thus there was no continuity between the loop and its corresponding wire below.  Tightened the nuts and secured the loop and there is no good continuity.  Dome opens and closes as it should using the handheld.  Due to inclement weather, mentor placed the system and hold with dome closed.  RTS at 1600 GMT.

12/16/13

MS

Mentor could not remotely open hatch on the dome of the OCO, Used hand held remote to open hatch, mentor requested system be started.  System is up and running at 1723 GMT.

12/12/13

MS

At 1516 GMT OCO hatch wasn't working correctly and was open in overcast conditions.  Manually closed hatch and contacted mentor about dome not operating correctly. OCO is in standby at this time. 

11/27/13

KT

The hatch was not responding. At 1440 GMT placed the program in hold and used the handheld and opened the OCO hatch. Reinitialized the program and the system began normal operations. RTS at 1455 GMT.

9/23/13

KT

Instrument mentor reported having issues with the new Sun Tracker that was installed and needed assistance through teleconference to try and troubleshoot the issues. At 1645 GMT attempted to fix the fluttering that was being produced, after several attempts we were able to get a nice steady beam of light. Next we attempted to adjust the instrument to align with the sun light that was coming down the shaft. After several attempts we were able to line up the sun light equally with the inlet tube. Instrument mentor still noticed a few issues with the instrument and was going to do more trouble shooting and diagnostic checks before going further. Instrument is online and operational at this time. RTS at 1841 GMT.

8/12/13

KT

Instrument mentor stated that the dome was stuck in the home position. While troubleshooting it was found that one of the 2 copper loops was broken and needed to be replaced. At 1626 GMT removed and replaced the broken copper loop then used the hand held controller and placed the dome back into the home position and called instrument mentor to let her know that the part was replaced and the instrument was ready to begin normal operation. At 1640 GMT Instrument mentor placed the instrument into play state. The hatch opened and the dome rotated to the correct position for this time of day and began normal operations. RTS at 1646 GMT.

7/31/13 - 8/1/13

CAM

The suntracker computer hard drive failed.  Possibly as a result, the suntracker had driven past the single limit switch and continued on, bending the other 3 switches arms over backwards and wrapped the data cable tight around the pedestal.  Powered suntracker electronics and computer off.  Removed computer to lab and replaced hard drive and CD-Rom drive.  Installed freedos on new drive.  Copied autoexec.bat and config.sys files from backup disks to new drive.  Copied suntracker program and config file from backup disks to new drive.  Edited suntracker config file for correct latitude, longitude and offsets.  Tracker azimuth drive tensioner arm had pulled loose when the tracker cable got wrapped tight around the pedestal.  Reattached tensioner arm.  Straightened bent limit switch arms.  Performed suntracker zeroing procedure and fine-tuned to center light path through tube.  Saved new azimuth and elevation offsets (-1.5 and 111.95).  Restarted suntracker and IFSloop program.  RTS at 1410 GMT.

7/31/13

KT

Instrument mentor requested that the UPS be removed and replaced. At 1335 GMT powered down the system and removed and replaced the UPS provided by instrument mentor. Powered system on at 1350 GMT. The Sun Tracker computer would not boot up due to hard drive failure. Trouble shooting still in progress on the Sun Tracker computer.

7/22/13

KT

Instrument mentor requested the change out of the OCO storage drive. At 1403 GMT changed out the OCO storage drive. After change out was completed I called instrument mentor to let her know that the change out was completed. RTS at 1418 GMT.

7/15/13

JEM

The IFS computer power supply fuse blew on 7/14/13 when storms moved through the area.  R/R the fuse @1405 GMT.  Tested the UPS to ensure that it was working properly and it was.  Reinitialized the system and RTS @1430 GMT. 

7/12/13

KT

Instrument mentor stated that she was not able to talk to the instrument. At 1620 GMT found that the fuse located in the IFS Power Control box had blown due to several power bumps that occurred on site. Removed and replaced the blown fuse and powered the system on. At 1625 GMT system began normal operations. RTS at 1635 GMT.

7/2/13

KT

The dome was found in the home position and the hatch was closed. Error message on the HERC computer stated instrument fail. At 1430 GMT stopped the program. Used the hand held to test the dome for rotation as well as to open the hatch. Both the dome and hatch responded. At 1439 GMT initialized the program. at 1445 GMT the OCO was online and operational.

6/26/13

KT

Instrument mentor was not able to communicate with the instrument. At 1900 GMT began trouble shooting with the instrument mentor via phone to figure out what the issue might be. After further inspection it was found that the Instrument Power Control fuse was blown and needed to be replaced. Removed and replaced the blown fuse. RTS at 1940 GMT. 

6/21/2013

KT

Instrument mentor requested that the OCO drive be swapped out. At 1400 GMT swapped out OCO drives. Took old drive down to shipping so it can be sent back to mentor. RTS at 1415 GMT.

6/19/2013

KT

Instrument mentor requested that the CaF2 window be cleaned. At 1800 GMT stopped the program and by using the handheld placed the dome into position to be able to access and clean the window. After the window was cleaned the program was started and the instrument began normal operations. RTS at 1820 GMT.

6/19/2013

KT

Instrument mentor stated that attempts to put the dome into the home position failed. The dome was found facing West and the hatch was opened. Attempted to put the dome into home position and the dome just rotated around in circles. Stopped the program and used the handheld and manually placed the dome into position so that the copper loops could be checked as well as all other cables and connections. Started the program and the dome acted like normal and found the position where it needed to be. RTS at 1430 GMT.

5/9/13

KT

The Dome was found facing Northwest with the hatch open. It appears that the system locked up not letting the dome rotate back to home and allow the hatch to close. During inspection, James Martin and I inspected the cables and wiring inside the dome housing. Found one connector that needed to be cleaned. After cleaning was completed the command was given for dome home. The dome rotated to the home position then rotated to align with the sun. RTS at 1420 GMT.

3/19/13

KT

The OCO hatch was open but the dome was facing Northwest. At 1550 GMT typed Dome Home command which rotated the dome back to the home position. Then dome rotated to the correct position and began normal operations. RTS at 1605 GMT.

3/18/13

CAM

Instrument mentor called and indicated the hatch would not open or close.  Placed dome in home position and confirmed both plates were making contact with the copper loops.  Rotated the dome to better access the loops for inspection and found that one of the nuts securing the electrical connection on the underside was loose.  Tightened the nut and tested hatch operation.  Hatch moves freely in both directions.  Notified mentor and she restarted the software.  RTS at 1740 GMT.

3/5/13

PAD/KT

OCO container heater failed.  Temporary heaters have been placed in container. Removed HVAC unit and took to electronics lab to troubleshoot.  Maytag model MEY12F7E, SN MR 154145 344Y.  RTS at 2100 GMT.

2/27/13

PAD

The site lost power at approximately 0350 GMT on 2/26 due to a severe winter storm.  At 1600 GMT, shut down Herc computer, disconnected Herc power supply, shut down IFS controllers, suntracker power supply and computer.  Power was restored at 1920 GMT.  On 2/27/13, charged external batteries for Herc computer.  Installed batteries at 1940 GMT.  Worked with Mentor to power up Herc computer, IFS controller, Suntracker computer and electronics box.  All systems returned to normal.  Too windy for hatch to open but manually verified hatch is operational.  Shelter heater has failed.  After start up, room temperature is 10C.   Installed two temporary electric heaters in shelter and brought room temperature up to 20 C.  RTS at 2020 GMT.

1/8/13

PAD/KT

Recently dome is having trouble moving into certain azimuth positions.  Mentor requests hatch azimuth drive motor replaced with spare previously sent to SGP. Shut down program at 1950 GMT.  R/R dome azimuth drive motor.  New motor installation required drilling two holes in fiberglass to access screws that held spring loaded bracket for motor drive shaft in place.  Two new mounting bracket holes were also drilled to accommodate new motor bracket.  Installed new motor, checked for free movement of azimuth drive, and initialized program.  RTS at 2120 GMT.

12/26/12

PAD/KT

Received email that OCO dome had failed to open.  At 1446 GMT, observed hatch was 1/2 open.  Hatch would open/close with manual controller.  Exited and restarted IFSloop.  Dome opened but struggled to turn to correct azimuth position.  Mentor called and said dome was having trouble going past home position to track sun in the afternoon.  At 1946 GMT, stopped program and checked voltages going to azimuth and hatch motors. 4-4.5VDC and 9-10.5VDC respectively.  Covered mirrors to prevent over spray and lubricated bearings on six rubber rollers that support dome.  Adjusted brass contact loops to not drag as hard when dome is at home position.  Checked amperage drawn on OCO monitor when hatch is moving or dome is adjusting azimuth position.  Both showed spikes of up to 15A at different points.  Restarted program.  RTS at 2126 GMT.

12/24/12

KT

The program was found in idle mode and the hatch was not open. Worked with instrument mentor to get system online and get the hatch open. System began normal operations.

12/21/12

JEM

The instrument mentor sent an email indicating that he had gotten the OCO up and running but the hatch was having trouble opening and closing on the dome.  Opened the dome and inspected the copper loops @1740 GMT.  One of the loops had become flattened and the loop was only intermittently making contact causing the hatch problems.  Reshaped and reinforced the copper loops.  The loops may require replacement but for now the OCO is working normally.  RTS @1750 GMT.

12/21/12

KT

The hatch was not opened. Tried opening the hatch with the remote control and hatch would still not open. Went topside and wiggled hatch then tried opening with remote, no response. While checking the computer the computer rebooted due to a possible wrong key being pressed. System came online but is now asking for login and password which are unknown and not written down. Emailed instrument mentor. RTS at 1600 GMT.

11/20/12

CAM

Local electrical service was shut off at 1410 GMT so that Kay Electric could upgrade its power lines from 7.2 KV to 14.4 KV in Grant County.  OCO was gracefully shut down at 1330 GMT in anticipation of power outage.  Power was restored at 2115 GMT after electrical upgrades were complete.  Restarted OCO and RTS at 2210 GMT.

11/16/12

KT

Instrument mentor requested that the metal plate be removed and replaced inside the dome. Placed dome in hold mode. Removed and replaced metal plate. Initialized dome. Instrument running and performing normal operations. RTS at 2015 GMT.

11/16/12

KT

Observed that the Suntracker was not tracking the sun. At 1640 GMT closed the dome, stopped and restarted computer. At 1647 GMT dome opened up in correct position and began normal operations. RTS at 1655 GMT.

10/24/12

KT

The OCO dome hatch was open during high winds. Dome close failure appeared on computer.  Used lubricant and sprayed the tracks that the hatch slides across while opening and closing. Adjusted slide clips to allow more movement. RTS at 1655 GMT.

10/17/12

CAM

Swapped data disk at 2205 GMT.

8/29/12

PAD

Placed instrument in hold mode at 1849 GMT to check roll pin on hatch drive motor.  After restarting program, hatch did not open fully but showed open position.  New indicator switches for hatch door were recently purchased and need to be installed.  R/R dome hatch limit switches.  RTS at 1934 GMT. 

8/20/12

PAD

Program shows failure.  Log shows vacuum pressure failure at 1212 GMT.  Vacuum shows .99 hPa.  Exited/ restarted IFSloop program.  Program started correctly.  RTS at 1342 GMT.

6/26/12

PAD

Notified by email OCO dome had failed to open.  At 1318 GMT, observed failure shown on monitor.  Restarted program and dome still failed to open.  Able to move hatch door with manual control.  Opened hatch and cleaned brass contacts located on base of dome.  Closed hatch and initialized program.  Hatch opened, dome aligned with sun, and instrument started taking measurements.  RTS at 1333 GMT.

6/21/12

PAD

Notified by mentor that comms have failed to OCO computer.  Unable to ping OCO computer from on site.  At 1800 GMT, indicating lights on wireless transceiver at OCO building is showing link with transceiver located at Optical trailer.  OCO computer and phones are showing activity on network hub located in OCO shelter.  Power cycled transceiver at OCO building with no change in status.  Contacted network administrator.  Power cycled both transceivers with no change.  Further troubleshooting found transceiver at Optical trailer could be pinged but OCO shelter transceiver could not.  Removed antenae cable connector below transceiver at Optical trailer and found water in connection.  Removed water with canned air and reassembled.  Comms between transceivers were reestablished.  RTS at 2030 GMT.

6/18/12

PAD

R/R data disk at 1549 GMT.

5/9/12

PAD

Swapped data disks at 1618 GMT.

5/1/12

PAD

Received email notification OCO dome failed to open.  At 1316 GMT, was able to open dome hatch and close using manual controller.  Shut down/ initialized program.  Dome failed to open and program stopped.  Opened dome using manual controller and cleaned dome brass contacts.  Exited/ started program.  Dome opened correctly and instrument started taking measurements.

4/17/12

PAD/KT

OCO window at bottom of light tube coming into OCO enclosure needs cleaned.  Placed instrument into hold mode at 1937 GMT.  Opened hatch and accessed window.  Cleaned window with distilled water and lens paper.  RTS at 1957 GMT.

4/12/12

PAD

Swapped data disks at 1429 GMT.

3/12/12

PAD

Swapped data disks at 1526 GMT.

2/16/12

PAD

Notified by mentor dome is not responding to commands and will not move.  Found dome at 355 deg with hatch closed.  Dome would sporadically move then stall.  Manually assisted dome to home position and opened hatch.  Troubleshooting found roll pin had fallen out that connects dome drive motor to drive shaft.  Installed roll pin and started measurements.  Motor will now drive dome but seems sluggish.  Notified mentor.  Mentor will order replacement drive motor and send to be installed.  RTS at 1716 GMT.

1/23/12

PAD

Dome shows position of 134 degrees Az but it is closer to 0 deg.  Exited/ restarted program.  Dome moved into correct position after restart.  RTS at 1557 GMT.

1/9/12

PAD

Swapped data disk at 1615 GMT.  RTS at 1630 GMT.

10/5/11

PAD

Power supply for heaters located inside IFS enclosure and in dome has failed.  Replacement fuses were ordered but when installed, fuse immediately blew.  Disconnected output power wires from power supply and replaced fuse again.  Fuse immediately blew.  Mentor sent replacement power supply and it needs to be installed.  R/R BK Precision DC power supply.  Connected output wires and set power supply to 15V DC.  Current draw is 15 amps.  Old Model 1689, SN A930002279.  New Model 1689, SN A930003061.  RTS at 2205 GMT.

10/4/11

PAD

Swapped data disks at 1830 GMT. 

9/26/11

PAD/JEM/CM

Notified by mentor and email error messages suntracker had stopped during the weekend.  At 1445 GMT, entered OCO enclosure and could smell burnt electronics.  Troubleshooting found 5W resistor on board that was installed in solar tracker power supply on 9/22/11 had gotten extremely hot.  Board was removed and taken to electronics lab for repair.  Troubleshooting on Az drive motor found motor is still dragging, making a scraping noise, when turning clockwise even when disconnected from solar tracker Az drive.  Consulted with mentor and replacement Az drive motor will be ordered from vendor.  Mentor also notified BK power supply for system heaters had failed on 9/24/11 causing IFS temperature errors.  Troubleshooting found power supply main fuse blown.  We do not have replacement fuse at CF at this time.  Mentor will send replacement fuse and spares.  Waiting on repairs and instructions at this time.  RTS at 1600 GMT.

9/22/11

PAD/JEM/CM

1530 GMT.  OCO suntracker had failed on 9/3/11.  Troubleshooting had found 5W resistors on suntracker power supply board had been extremely hot and had melted some tracers on back of board.  Board was repaired in electronics lab and mentor had also ordered a new board from vendor.  Troubleshot possible Az drive motor failure by applying 12V DC directly to motor.  Motor had been able to turn Az counterclockwise but was unable to turn clockwise with suntracker power supply.  Applying 12V directly did not turn motor clockwise the first time power was applied but motor did start working after that.  Reconnected suntracker power supply but it would not power motor after initializing system.  Replaced suntracker power supply board with new board provided by mentor.  Powered up power supply and suntracker computer.  Suntracker initialized and both Az and Elevation mirrors will now move.  Mentor later called and I monitored system while she started OCO measurement using HERC computer after pumping down vacuum in instrument enclosure.  System started correctly and is now taking measurements. Light beam coming inside building is not in correct alignment indicating Az and/or Elevation offsets need to be adjusted.  Too late in the day to start adjustments.  RTS at 2030 GMT.

9/7/11

PAD

Solartracker failed on 9/3/11.  Mentor request voltage checked at electronics box for azimuth motor.  Found -1.5VDC for Azimot + and -, and +6.0 and -6.0 VDC for POLMOT, (elevation) + and -.  Troubleshooting found burnt resistors on power stage board in back of electronics box.  Took pictures of board and sent to mentor.  Waiting on mentor to decide whether to send board for repairs or send resistors to be installed on site.  RTS at 1800 GMT.

9/6/11

PAD

Notified by mentor via email suntracker had failed on 9/3/11.  Mentor had stopped program and shut down suntracker computer at that time.  At 1300 GMT, inspected suntracker limit switches and cables to verify suntracker had not bypassed limit switches or damaged cables.  No problems found.  Later worked with mentor trying to restart system.  Azimuth drive is not responding when system initializes.  Checked power supply in suntracker electronics box, cable connectors, computer card, etc.  Troubleshooting at this time.

9/1/11

PAD

At 1453 GMT, observed dome rotating between approximately 340-355 deg.  Display shows position rotating between 95-116 deg.  Stopped and reinitialized program.  Dome moved into correct position.  RTS at 1508 GMT.

8/25/11

PAD

Dome is giving “failed to close” errors.  Mentor requests brass contact loops checked.  At 1533 GMT, put instrument in hold.  Cleaned and reshaped brass loops and put in new Styrofoam inserts.  While system was in hold, cleaned window at bottom of light tube.  Reinitialized program and instrument started taking measurements.  RTS at 1615 GMT.

8/6/11

PAD

Notified by facilities on 8/5/11 via mentor that dome had failed to close.  A small rain shower had move through.  Consulted with facilities and was able to close dome hatch.  On 8/6/11 at 1415 GMT, TS found roll pin had fallen out that connects Azimuth drive motor to drive shaft.  Removed motor, found roll pin and reinstalled.  Azimuth drive is now working correctly.  Started program and notified instrument mentor.  RTS at 1515 GMT.

8/9/11

PAD

CF lost power on 8/8/11 at 2247 GMT.  Power was restored on 8/9/11 at 0211 GMT.  At 1646 GMT, found HERC and suntracker computers, and instrument active.  Opened hatch to verify suntracker was still operational and had not bypassed limit switches and wrapped up cords.  No problems found.  Monitor shows room temperature at 32.7C.  Started AC unit and set at 70 F.  Powercycled network hub and mentor can now remotely access computer.  Log file shows:  Entering State Bad_IFS Temp.  IFS temp is 31C.  Mentor started program and IFS temperature should slowly decrease as enclosure is cooled.  RTS at 1716 GMT.

7/25/11

PAD

At 1557 GMT, display shows dome moving between 103-107 deg Azimuth every 7 seconds.  Placed system in hold status and then reinitialized.  After startup, dome moved between 105-109 deg Azimuth several times then started taking a solar measurement.  RTS at 1612 GMT.

6/28/11

PAD

Monitor indicates dome is moving between 89-105 deg Az.  Actual locating is between 340-355 deg.  Placed system in hold and then reinitialized.  Dome failed to open due to high wind speed.  RTS at 1548 GMT.

6/22/11

PAD

OCO hatch is occasionally having trouble closing and opening.  Stopped program at 1809 GMT.  Lubricated hatch drive chain, tightened one loose bracket on copper contact loop, and checked hatch rail guides.  RTS at 1824 GMT.

JEM

6/10/2011

The OCO disk drive needs to be replaced.  R/R the OCO disk drive @1530 GMT.  Gave the old drive to the shipping department to be returned to the mentor.  Old drive - QNX4 #3   New drive - QNX4 #7  RTS @1545 GMT.

5/4/11

PAD

Swapped data disks at 1521 GMT.  RTS at 1536 GMT.

4/7/11

PAD

Mentor requests pictures taken of optical encoder located in IFS enclosure.  Mentor remotely opened valve to remove vacuum and allow instrument enclosure to equalize barometric pressure.  Opened enclosure and took pictures of optical encoder.  Mentor determined instrument will be shut down until next week when repair and maintenance is scheduled.  RTS at 1845 GMT.

3/9/11

PAD

R/R data disk at 1656 GMT.

1/31/11

PAD

R/R data disk at 1618 GMT.

1/18/11

PAD

Power bump on 1/15/11 had shut down enclosure heater.  Reset heater to 69F at 1705 GMT.  Room temperature was 12.2C at that time. 

12/20/10

PAD

R/R data disk at 1648 GMT.

12/1/10

PAD

AC power to CF will be shut down today for ongoing construction.  Instrument needs to be stopped until power is restored.  Shut down instrument at 1420 GMT.  Restarted at 1845 GMT.

11/22/10

PAD

AC power will be interrupted for construction near the main compound.  Need to gracefully shut the instrument down prior to power outage.  Powered instrument down at 1430 GMT.  Restarted instrument after AC power was restored.  RTS at 2000 GMT.

11/18/10

PAD

AC power needed to be secured to drill pier holes for new construction near the RCF trailer.  Need to gracefully shut the instrument down prior to power outage.   Powered system down at 1440 GMT.  Restarted system after AC power was restored.  RTS at 1820 GMT.

11/10/10

PAD

AC power will be interrupted for construction near the main compound.  Need to gracefully shut the instrument down prior to power outage.  Powered system down at 2158 GMT.  Restarted system after AC power was restored.  RTS at 2335 GMT.

10/28/10

PAD

Program shows hatch position is 126 degrees azimuth.  Actual position is 10 degrees.  Placed program in hold mode and then reinitialized.  Hatch moved into correct alignment with sun. 

10/13/10

PAD

Swapped data disk at 1601 GMT.

9/27/10

PAD

R/R data disk at 1444 GMT.

9/8/10

PAD

OCO enclosure air conditioner lost its setting due to power bumps on 9/7/10.  Reset air conditioner to 69 F at 1248 GMT. 

8/30/10

PAD

Window in bottom of tube located in base of dome is dirty and needs cleaned.  Placed system in hold mode at 1901 and cleaned window.  Restarted system at 2001 GMT.

7/30/10

PAD

At 1503 GMT, dome position was indicated as 89 deg Az, while actual position is ~50 deg.  Stopped/ reinitialized program and dome move into correct position. 

6/7/10

PAD

Notified by mentor on 6/5/10 dome was having trouble finding its home contacts. At 1555 GMT, found loose bracket that holds brass contact on dome.  Tightened bracket and moved dome into home position.  Shutter status is unknown.  Initialized program, shutter status would briefly show closed, but then program would go into failure mode.  Exited/ restarted program with IFSloop.  Program started correctly, shutter status shows open, and dome moved into correct alignment.  RTS at 1615 GMT.

5/11/10

PAD

Swapped hard drive disks at 1517 GMT.  RTS at 1532 GMT.

4/29/10

PAD

OCO UPS, model PW9120-1500/ PN 05147358-5501, that supplies power to the IFS computer, suntracker computer and control box, and laser control box fan has failed.  At 1525 GMT, shut down system according to mentor’s instructions and replaced with spare UPS.  Restarted system and notified mentor.   Provided mentor with cooling fan specs so a replacement could be ordered.  RTS at 1550 GMT. 

4/19/10

PAD

Received OCO error message:  Vacuum failure.  Indicated by pressure or fault in pump power latch.  At 1258 GMT, IFS_P = 1.43 hPa.  Exited/ restarted IFSloop program.  Program started correctly and instrument was put in nap state due to high humidity.  Notified instrument mentor.  At 1441 GMT, observed instrument operating correctly.  RTS at 1318 GMT.

4/7/10

PAD/CM

Received email error message stating hatch had failed to open.  At 1245 GMT, TS found loose connection on wire coming from bottom hatch limit switch at block located in top of hatch.  Tightened wire and hatch will now close.  Exited program and restarted.  Hatch opened and dome moved into correct position.  RTS at 1305 GMT. 

4/5/10

PAD

Notified by mentor hatch is sometimes having trouble opening and closing.  Too windy to troubleshoot.  Put program in hold at 1420 GMT until wind speed decreases.

3/30/10

PAD

At 1411 GMT, observed dome was approximately 15 deg Az.  Monitor shows dome at 98.7 Az.  Put program in hold and then initialized program.  Dome moved into correct position.  RTS at 1426 GMT.

2/24/10

PAD

At 2003 GMT, put program in hold and installed Styrofoam supports on brass contact loops in hatch.  This will hopefully prolong the life of brass loops and help ensure contact between loops and metal plates when hatch is at home position.  Restarted program.  RTS at 2023 GMT.

2/24/10

PAD

While doing PM rounds at 1647 GMT, observed Suntracker computer shut down.  Error in log file on Herc computer:  Max syntax error count reached.  Can’t connect to 192.168.1.101:80.  There were 78 seconds left in the scan instrument was currently performing.  Put instrument into hold and tried to initialize.  Program would not start.  Exited and restarted program.  Program still would not restart.  Contacted mentor and troubleshooting found instrument computer and controller, both located under instrument, had also shut down.  Later contacted by mentor explaining a software glitch had shut down both suntracker and instrument computers.  She reset the software at 1810 GMT, Observed instrument was taking measurements correctly. 

2/17/10

PAD

Notified by mentor OCO dome has been having trouble finding home position.   Troubleshooting found broken brass loop on hatch that makes contact when hatch is in home position.  R/R broken brass loop.  RTS at 1732 GMT.

2/4/10

PAD

R/R data disk at 1438 GMT.   

1/31/10

PAD

Notified by CF via instrument mentor OCO hatch will not close and freezing rain is forecasted.  At 2000 GMT was unable to close hatch using manual controller.  Troubleshooting found hatch was stuck in open position due to recent winter storm.  Broke hatch free and was able to close hatch using manual controller.  RTS at 2025 GMT. 

12/30/09

CAM

Instrument mentor advised she could not access the OCO computer.  Checked status in building and found the computer was running as it should, but the VoIP phone indicated there was no network connection.  Tried power cycling the network bridges with no change in status. Removed external antenna connections and found water inside at both the master and remote antennas.  Used canned air to dry connections.  Rebooted bridges and link was reestablished at 1615 GMT.

12/7/09

PAD

Requested by mentor to swap computer external hard drive.  R/R hard drive at 1712 GMT. 

10/20/09

PAD

Notified by mentor hatch is sometimes having issues finding home position.  Mentor requests brass contact strips checked for damage or in need of cleaning.  At 1429 GMT, R/R broken brass contact strip.  RTS at 1454 GMT.

9/21/09

PAD

At 1421 GMT, observed hatch was open but not aligned with sun.  Display shows hatch at 105.03 degrees Azimuth.  Actual position is approximately 45 degrees.  Put program in hold mode and then initialized program.  Program started and hatch moved to 100.33 degrees in correct alignment with sun.  RTS at 1436 GMT. 

9/4/09

PAD

At 1847 GMT, exited program.  Mentor requested power supply in solar tracker control box replaced.  R/R power supply as instructed.  Switched input power to control box from raw power to UPS as instructed.  System restarted correctly and started taking solar measurements.  RTS at 1927 GMT. 

9/9/09

PAD

At 1523 GMT, found hatch closed.  Log file show at 1213 GMT: Laser period too slow or modulation too small.  At 1243 GMT: Entering IFS_Status_Failing.  This message was repeated at 1335, 1416, and 1525 GMT.  Exited program and initialized IFSloop.  Program started correctly and instrument started taking solar measurements. 

8/19/09

PAD

At 1522 GMT, display shows hatch at 111 deg azimuth, but actual bearing is approximately 310 degrees.  Put program into hold then reinitialized.  Hatch shown alternating between 100-114 degrees but hatch is actually moving between 50-65 degrees azimuth.  Exited program then restarted IFSloop.  After start up, hatch was alternating between 98-116 degrees azimuth and appears to be in correct position.  RTS at 1537 GMT.

8/5/09

PAD

At 1446 GMT, observed hatch open but not aligned with sun.  Scrolling status shows hatch rotating between 91-99 degrees.  Actual position is approximately 45 degrees.  Put instrument in hold mode then reinitialized.  Hatch opened and moved to correct alignment.  RTS at 1501 GMT.

7/30/09

PAD

At 1428 GMT, observed dome is open and in home position.  Solar tracker computer shows N/C, no connection, with tracker.  It also shows “hardware error” at bottom of screen.  Herc computer status shows: Suntracker Flip 0. This is repeated.  Solar tracker has single limit switch tripped.  Initialized solar tracker and it stopped at 186 degrees Azi, 38 degrees Elevation.  Mentor called at this time and suggested rebooting solar tracker computer and then exiting and restarting IFSloop on Herc computer.  Program started correctly and solar tracker is in correct alignment.  Mentor suspects power supply for solar tracker is failing.  RTS at 1453 GMT.

7/21/09

PAD

Received OCO email status indicating a vacuum failure.  Program had gone into failure mode.  At 1352 GMT IFS_P was 5.06 hPa.  Manually pumped down instrument enclosure and restarted program.  RTS at 1412 GMT. 

7/20/09

PAD

Solar tracker has been out of adjustment and instrument shut down since 7/16/09. See related technotes for more details.  At 1827 GMT, set zero azimuth and offsets according to instructions.  Azimuth old 4.40, new 4.25 degrees.  Zenith old 112.20, new 112.0 degrees.  Shutter would not open after adjustments.  Removed shutter and adjusted brass straps to connect when dome is in home position.  Hatch and shutter are now working properly and solar tracker is in correct alignment. RTS at 2101 GMT.    

7/17/09

PAD

At 1417 GMT attempted to set 0 on solar tracker as instructed in manual.  After manually holding limit switches down, tracker initialized and appeared to have found 0 but is not in correct alignment after program start up. Tracker will stop at different points it thinks is 0 depending on when limit switches are held down after being tripped by tracker.  Trying to reset 0 according to manual produced same results.  Tracker will not initialize without having limit switches manually held down. Shut instrument down for the weekend at 2100 GMT.  Troubleshooting at this time.

7/16/09

PAD

At 1513 GMT, observed shutter open but sunlight is not coming into enclosure.  Solar tracker monitor shows NC.  Power cycling computer to reinitialize solar tracker caused solar tracker main amp board to get very hot.  Further troubleshooting found tracker had gone past limit switch and continued until stopped by attached cable.  This was causing controller to overheat.  Mentor advised straightening out limit switches and manually moving tracker until it is in correct position.  Tracker failed to initialize on start up and shows:  Timeout/ abort.  No 0 pt. Found.  Still troubleshooting at this time.

7/8/09

PAD

Display shows hatch rotating between 150-158 degrees Azimuth.  Actual position is around 45 degrees.  Put instrument into hold and then restarted.  Hatch aligned into correct position with sun at 153 degrees.  RTS at 1821 GMT.

6/29/09

PAD

At 1412 GMT, observed OCO hatch facing approximately 315 degrees Azimuth.  Display shows hatch at 89.35 degrees.  Put program in hold mode and hatch returned to home position.  Initialized program and hatch moved into correct alignment with sun and shows 76.81 deg.  RTS at 1427 GMT.

5/26/09

PAD

OCO dome has recently sometimes displayed a different azimuth position than actual.  Mentor requests encoder wheel check for possible causes.  At 1429 GMT, found mounting bolts attaching encoder wheel assembly to dome were loose.  Found necessary hardware and tightened bolts.  RTS at 1519 GMT. 

5/18/09

PAD

At 1530 GMT, observed dome facing NE and it is not aligned with sun.  Computer shows dome at 94.06 degrees azimuth.  Dome is closer to 35 degrees.  At 1534 GMT, dome started moving between 94.06 and 111.3 according to display and indicated it was correcting alignment.  At 1536 GMT, put instrument into hold mode and dome came around close to home position but did not close hatch.  At 1537 GMT, dome moved to home position and closed hatch.  Started program at 1540 GMT and dome moved into correct alignment with sun.  Display shows 95.62 degrees.  RTS at 1545 GMT.

4/16/09

MK

Wind speed and direction are both showing 0.0.  Troubleshooting found ~1.8V coming from wind direction indicator.  This is within acceptable range and is fluctuating slightly with changes in wind direction.  With oscilloscope connected to wires from wind speed indicator at data logger box sine wave shows .7V.  Removed wire from data logger and it now shows 1.2V.  Reconnected wire and wind speed and direction is now displayed on monitor.  RTS at 1910 GMT. 

4/8/09

PAD

Removed BK Precision DC power supply, model 1689/ SN A93001367 used for heaters located in Bruker instrument enclosure.  Installed new power supply, same brand and model, SN A930002279. Sending removed unit to mentor for repairs.  RTS at 1855 GMT.

1/26/09

PAD

Freezing rain starting at CF.  Put program in hold mode and closed hatch at 1642 GMT.  Notified instrument mentor.   

11/24/08

PAD

Notified by mentor hatch is not tracking sun in the afternoon hours.  Mentor suspects something causing hatch to bind up that is not allowing hatch to rotate freely.  At 1930 GMT, troubleshooting found bolts on plate covering position sensor switch on hatch base were loose and plate was binding on hatch.  Tightened bolts but hatch is still binding.  Further troubleshooting found mounting bolts on hatch drive motor were loose allowing motor and drive gear to slide down causing drive gear to bind up on ring gear in hatch.  Moved motor assembly up into correct position with drive gear centered in ring gear.  Hatch will now turn freely.  Notified instrument mentor.  RTS at 2030 GMT.

11/18/08

PAD

At 1557 GMT, stopped program to pull vacuum on OCO.  After restarting program status said:  Failure/ Dome close shutter/ Power pump off.  Stopped/ restarted program with same results.  Left voicemail with mentor who later called and said she suspects problem with hatch, which did not close completely after stopping program.  Mentor exited completely out of program then restarted and program started correctly.