Instrument Log  

SACR             C1       Lamont, OK

 

Date

Technician

Comments

7/22/16

DKB

Found the system UPS off due to high shelter temp 7/21/16 2030 GMT. AC unit seem OK. Powered up the system at 1400 GMT. Powered off the Ka transmitter and PDU for AC unit cleaning at 1510 GMT.

7/14/16

DKB

During morning checks found the UPS powered off. Powered on and restarted the system at 1345 GMT. Unknown reason for UPS failure as it was well before storms overnight.

6/28/16

DKB

Found system inoperative with temp errors and chiller 1 Off. Visual inspection no problems noted. Logged on to local terminal and Chiller powered up. Powered up the PDU and restarted the transmitter to ready at 1352 GMT. Also noticed the new chiller is reporting input and output temps but not the watlow reading. Chiller 1 is running at 20.0 C.

6/27/16

DKB

Chiller 1 is excessively noisy and the output pressure is jumpy. It seems as the pump may be going bad. Replaced with spare but spare would not power up. T/S with vendor found that this unit is configure for DC control signal to power up (Not required for our system). Reconfigured and restarted the Ka system at 2040 GMT. Old s/n – 10080331, New s/n - 12080536 WD76542

6/9/16

BMA

Mentor requested we remove the RDS computer from the WSACR for use in KAZR test system. RDS removed and WSACR left powered down for duration of testing. RTS 6/9/16  1930 GMT.

6/3/16

BMA

Mentor requested W band LNAs installed. LNAs installed. S/N – H 18244001005, V 18244001002 RTS 1930 GMT.

5/26/16

BMA

Mentor requested W LNAs be removed and shipped to him for testing. LNAs removed and shipped. Sn: 16997001006,16997001001 RTS 5/26/16 1900 GMT.

 

5/25/16

DKB

Found both PDU's powered on but down with no faults. Both transmitters are down. Cycled the power to both Radar 1 and 2, powered on the PDU outputs, restarted the transmitter SW, and tried to restart both transmitters. Both the Ka and W transmitters would only come up to ready at 1331 GMT. Mentor maybe working on the system.

 

5/17/16

DKB

Found PDU #2 powered off since 1200 GMT. Powered on the PDU, powered on the W Band transmitter power supply, server, modulator, and heater, restarted Radar 2 SW and transmitter to Ready at 1404 GMT. Tried to power on RF but it would only stay in ready mode. Mentor maybe working on the system.

5/16/16

BMA

While doing daily checks, noticed Radar 1, Radar 2, and Chiller 1 were off. Restarted equipment and software. RTS 5/16/16 at 1545 GMT.

5/11/16

BMA

Mentor requested we swap data drives on RDS2 because he could not detect the installed drive. 1TB drive swapped for a 2TB drive. RTS 5/11/16 2000 GMT.

4/20/16

BMA

Mentor requested pedestal data readings be taken to ensure data was being transmitted to RDS. Readings taken. Data is reaching RDS. Shutdown: 4/20/16 1830 GMT; RTS: 4/20/16 1915 GMT

4/15/16

DKB

Chiller #1 is shutting down. Found chiller 1 powered off and restarted. The temp reading was 16.2C and the fan was off. Once the temp reached 18C the fan started and stayed running. Powered up the PDU's and watched the temp rise to 20C and stabilize. Powered on both the Ka and W transmitters and both system were radiating at 1428 GMT.

4/13/16

DKB

Found the system shut down at 1100 GMT with high temp. Restarted both chillers and no problems found during the daily checks. Chiller 1 shut down itself and was restarted at 1421 GMT. It failed again and despite no problems found but it shut down later in the date.

4/11/16

DKB

Found both chillers powered off and restarted at 1415 GMT. Both transmitters were also off due to chiller failure.

4/7/16

BMA

Mentor requested dongle be removed and DSA testing be conducted on KaSACR. It appears that removing the dongle has fixed the problem. DSA testing postponed until we receive further guidance from mentor. Shutdown: 4/7/16 1800GMT; RTS: 4/7/16 1845 GMT

3/22/16

DKB

Powered off both chillers and replaced the water filters and fan filters. Added coolant mix to top off reservoirs. Water pressure after filters 42/ 38 psi. RTS 1533 GMT.

12/22/15

BA

Antenna would not respond to movement inputs through MCC.

Justin Bullock from Orbit determined that the DSAs were addressed wrong and that the elevation DSA was bad. DSA was replaced and addressing problem corrected.

12/15/15

BA

Chiller 2 in the SACR was leaking from the pressure gauge.

Replaced the chiller with a spare unit. The new chiller failed to operate properly. Took gauge out of new chiller and put it in old chiller and reinstalled.

11/25/15

BA

Az and El DSAs were not responding to inputs.  Techs troubleshot with Justin Bullock from Orbit. We were able to correctly reprogram the DSAs but they will not accept inputs from the MCC. Will continue troubleshooting.  RTS at 2100 GMT.

11/25/15

BA

Mentor requested to change Netgear gateway with Supermicro gateway.  Supermicro *gateway programmed and installed at 1700 GMT. 

*Old SN – 2CC10735002D5.  *New SN – S5FFAEH13510428

11/10/15

BA

Mentor requested that we change the serial card and dongle on the KaSACR.  Serial card and dongle replaced at 1600 GMT.

11/5/15

BA

SACR was powered down at 1500 GMT for power upgrade.

Power system was successfully upgraded by Hamelmenn Communications. RTS at 2100 GMT.

9/22/15

Mentor

RDS2 computer was pulled on 9/2/15 due to multiple problems.  The computer fan had failed as well as one on the receiver card.  The data hard drive had also failed.  Mentors on site this week for training and replaced RDS2 with a spare KAZR computer.  The OS hard drive and serial card were removed from the original SACR computer and installed in the replacement computer. 

*Old – BSI 1027722  *New - 1027727

9/15/15

KT

Coolant had leaked all over the floor from the chiller unit. At 1408 GMT cleaned up the coolant leakage. #2 chiller hose had been disconnected and powered off, and during further inspection Chris Martin had found that chiller #2 had been powered back on causing the coolant to leak through the quick connect fittings. RTS at 1423 GMT.

9/1/15

BA

Dehydrator showing Low Pressure and Humidity alarms.

Bypassed pressure regulator. Will continue to observe to see if problem is fixed.

8/31/15

CAM

Dehydrator was rebuilt on 8/21.  Dehydrator hours showed 12272 in maintenance panel.  Reset hours to 0 at 1900 GMT.

8/21/15

BA

The dehydrator had been removed for rebuild. Dehydrator was rebuilt. Will continue to monitor to make problem was fixed.

7/24/15

BA

System would shut down due to overheating.  Replaced HVAC-1. Will continue to monitor to ensure proper/satisfactory cooling of system.

7/2/15

BA

W-SACR computer is not receiving data from the pedestal.  Mentor sent replacement serial card.  Replaced RDS2 serial card and checked all cables.  Cables tested good.  Mentor suspects software issue and will discuss with vendor.

6/23/15

CAM

The top air conditioner is not cooling properly.  Removed the air conditioner from the shelter and used water hose to clean the coils.  Reinstalled air conditioner and checked operation.  Unit does appear to be cooling better now, but will need to see how it performs during the heat of the day.  Restarted Ka radar and RTS at 1523 GMT.

6/22/15

CAM

Found radar system had shut completely down due to shelter over temperature on Friday, 6/19.  Restored UPS load power at 1500 GMT.  The top air conditioner does not seem to be blowing as cold as the bottom unit.  By 1929 GMT, the shelter temperature had reached the upper limit of 33 degrees C and the system shut down again.  Will pull the top air conditioner tomorrow to clean the coils and see if its operation improves.  System remains down at this time.

6/18/15

CAM

Ka-SACR was not transmitting.  Started sgp_sacr_kaw_ops.pax and Ka resumed transmitting at 1348 GMT.

6/17/15

CAM/BA

The site experienced a one hour power outage at approximately 1000 GMT.  The radar system shut down automatically at 1012 GMT due to high shelter temperature.  Restarted Ka-SACR and RTS at 1420 GMT.  W-SACR still down pending replacement of serial interface.

6/16/15

CAM/BA

Ka-band radar started following maintenance and RTS at 2100 GMT.

6/16/15

CAM/BA

Unable to access W-band PDU through the web interface to start the W-band radar.  T/S found the network switch in the W-band RF enclosure had failed.  Removed Black Box PoE switch and replace with Tel Sync non-PoE switch.  Was then able to access PDU and power system up, but radar software would not run.  Mentor suspects failed serial interface card in RDS2 and will send replacement.  W-band radar down pending repairs.

6/16/15

CAM/BA

While replacing W-band noise source, observed significant condensation inside both RF boxes.  T/S found that the air flow regulators on the SDRAR box were completely closed so no dry air was getting to the boxes.   Opened regulators and confirmed air flow is now getting to RF boxes, however there appears to be an additional problem with the dehydrator not pumping air.  T/S of dehydrator to continue. 

6/16/15

CAM/BA

W-band channel 2 noise source returned from being repaired/calibrated and needed to be installed.  Installed channel 2 *noise source at 2000 GMT. 

*New - 17030001001

6/10/15

CAM

Power to all ground mounted transformers is being cut in order to connect the new power panel near the AOS trailer and run power to new PGS building.  Instrument needs to be shut down.  SACR was not currently transmitting due to known issues, but was powered down at 1325 GMT.  Restarted computers and electronics once power was restored, but left transmitters off.  RTS at 1545 GMT.

6/4/15

CAM

Power to all ground mounted transformers is being cut in order to upgrade the transformers at AOS trailer, IDP2 and 50 profiler pad.  Instrument needs to be shut down.  SACR was not currently transmitting due to known issues, but was powered down at 1305 GMT.  Restarted computers and electronics once power was restored, but left transmitters off.  RTS at 1815 GMT.

5/14/15

CAM

Instruments located in KAZR and SACR vans need to be shut down to allow contractor to install shelter grounding upgrades.  Shut down system at 1640 GMT.  Powered up system and had to reset RDS1 computer before it would boot up correctly.  W-band is currently not in service.  System was left with all system powered but radars not transmitting.  Notified Mentor.  RTS at 1825 GMT.

5/6/15

RSI

RF survey was done for Ka-SACR on 5/6 by RSI Corporation.  The findings were that the RF exposure at the GIF and optical trailer platforms was less than 4% of the MEL for the general population with the antenna stationary and transmitting in that direction.

4/15/15

CAM

Instrument mentor requested W-band channel 2 noise source be removed and returned to vendor for warranty repair.  Removed *noise source and isolator at 2030 GMT and delivered to S/R for shipment to vendor.  *Old - 17030001001

2/24/15

CAM/BMA

W-band High Speed Power Controller (HSPC) needs to be replaced due to problems with the noise source drivers.  Shut SACR down at 1500 GMT.  Pulled RF unit out and noticed that the noise sources were connected to the board opposite from the way the cables were labeled (J3 cable connected to J4 on HSPC, and vice-versa).  Contacted Jeff at ProSensing and he believes the cables were mislabeled and advised to hook the new controller up the same as the old one was.  R/R HSPC and restored power at 1615 GMT.  Turned transmitters on and RTS at 1630 GMT. 

2/19/15

BMA

Mentor requested replacement of W-band channel 1 *LNA on SACR. SACR was shut down at 1840 GMT and W-band channel 1 LNA (Model Number: QLN-94013520-11) was replaced. Restored power at 2125 GMT and turned transmitters on at 2150 GMT.

*Old – 16997001002  *New - 16997001006

1/12/15

CAM

Ka band radar was not transmitting.  Radar software was down.  Restarted Ka radar software at 1850 GMT.  Turned transmitter on and RTS at 1852 GMT.

1/7/15

CAM

W-band software would not run following maintenance on 1/6.  RDS2 root drive had filled up due to large log files being written.   Brian Simakauskas from ProSensing remotely cleared the large files and restarted software.  W-band RTS at 2200 GMT.

1/6/15

CAM/MG

Vendor provided additional items that needed to be installed to complete the upgrade efforts started in December.  Shut both radars down at 1955 GMT.  Installed RTD extensions for coolant lines and hygrometers in both RF enclosures and connected as requested.  Replaced 4” standoffs on lower rear radiators with 6” standoffs and secured in both enclosures.  Removed existing Ethernet cable between camera and Ka enclosure and capped Ka bulkhead.  Installed new Ethernet cable between camera and W enclosure and connected W bulkhead to switch.  RTD and ground connections between EIKA and modulator still need to be cut and fitted with Molex connectors to facilitate replacement of components.  This will be done at a later time as the RF unit must be removed to install connectors.  Restarted Ka radar and RTS at 2125 GMT.  W-band software would not restart.  Notified mentor.

1/5/15

CAM/MG

Vendor requests an additional radiator installed in W and Ka RF enclosures.   Shut W-band radar down at 1925 GMT (Ka-band was already down).  Installed additional radiator in W and Ka enclosures as per vendor’s instructions.  Restarted radars and RTS at 2050 GMT.

12/1/14 -

12/5/14

Mentor

Vendor

ProSensing (vendor) and mentor were on site week of 12/1 for repairs and upgrades.  W antenna mounting frame was modified and antenna lowered closer to enclosure.  RF enclosure racks were completely redone and RF units modified to make modulator replacement much easier.  RF plate layouts were also modified.  Ka flex waveguides were damaged and replaced with waveguides from AMF2 SACR, which is in the warehouse waiting to be sent to vendor for repair.  New waveguides are on order and expected to arrive this week.

**Note:  Multiple components were swapped (modulators, etc) during this visit, but not yet documented.  LNAs were also replaced with different make/model.

10/29/14

CAM

Instrument mentor advised the chiller was faulting when he attempts to start the Ka-SACR.   T/S found that chiller 1 fan had failed.  Replaced fan and tested ok.  Chiller powers on and cools to set point with radar off.  Turned radar 1 on and notified mentor it was ready for testing.  RTS at 2050 GMT.

10/28/14

KT

Instrument mentor reported a fault with the #1 chiller. At 1615 GMT talked with Chris Martin and informed him that the fluid level in the #1 chiller was between the full line and fill line. It was requested that fluid be poured into the reservoir to bring the level to the full line. After this was completed the instrument mentor was contacted. Instrument mentor then advised that he would log into the system for further observation. RTS at 1630 GMT.

10/10/14

CAM

Mentor was able to piece together some rigid and flexible waveguide pieces from his lab to replace the damaged Ka flexible waveguide.  Installed waveguide at 1600 GMT.  Notified mentor that Ka-SACR was ready for testing/operation. 

9/22/14

KT

During inspection of the SACR, coolant was found to have leaked from the chiller on to the floor. At 1525 GMT cleaned up leaked coolant and checked connections on the coolant lines. All connections appear to be tight. Filled the upper coolant holder. RTS at 1540 GMT.

9/23/14

CAM

Instrument mentor requested measurements in order to replace the Ka 12” flexible waveguides with rigid waveguide and then using just a short piece of flexible at the end.  Supplied measurements as requested.

9/16/14

 

CAM

Placed order for replacement 12” flexible waveguide pieces for Ka-SACR.  Vendor reports a lead time of 4-6 weeks.

9/15/14

CAM/JEM

Mentor requested that Ka-SACR channel 2 (cross-pol) LNA be replaced.  Powered radar1 and chiller1 off at 1800 GMT.  R/R *LNA.   Installed 0.1uF capacitor and RTD on new LNA prior to installation.  Inspection of flexible waveguide revealed possible damage to the corrugated liner of channel 2 waveguide.  Took photos and sent to mentor.  Radar was returned to ready state but left powered off at this time.  Notified mentor of possible waveguide damage and awaiting his recommendation. 

*Old – 1037  (A803504)   *New – 1089 (A16989) 

(part# LNA-28-01020)

8/29/14

JEM

Mentor indicates he is unable to log in to the MCC computer.  Found the MCC computer unresponsive.  Rebooted the MCC computer @1704 GMT.  Notified mentor that the computer is up.  RTS @1715 GMT.

8/29/14

JEM

The mentor reports, "The RDS1 PDU is not responding. Without the PDU nothing can be turned on inside the RDS1 RF unit."  Troubleshooting found that all 3 "white" fuses for RDS 1 in the PDU had failed.  Replaced all 3 fuses and RTS @1500 GMT.

8/28/14

CAM

Dry air system compressor motor failed on 8/11/14 and was taken in to be repaired.  Motor had to be replaced due to burned windings.  Installed new motor today at 2130 GMT.  Dry air system placed back into service.  Chillers and radars left powered off at this time to allow time for RF boxes to be dried.

8/27/14

CAM

Instrument mentor sent shock-isolation feet to be installed on the dry air compressor.  Installed shock-isolation feet at 1330 GMT.  Compressor and radar are currently down due to compressor motor failure.  New motor is on order.

8/11/14

CAM

All pressures reading zero on the dry air system, including the compressor tank.  The breaker for the compressor supply circuit had tripped.  Reset breaker but compressor still would not run.  Powered compressor off and locked/tagged out.  T/S found the start capacitor for the compressor motor had failed and the oil had leaked inside the motor.  Removed end piece from the motor frame and found the centrifugal switch had also failed and insulation had melted on much of the internal wiring.  Removed motor to be repaired.  Radar was not running prior to compressor failure and remains down at this time pending motor repair.

7/15/14

CAM

Instrument mentor requested RDS1 server be replaced with the spare server.  Powered Ka-SACR down at 1240 GMT and removed *RDS1 from RF enclosure.  Removed memory, hard drives and serial card from RDS1 and installed in the spare server.  Connected keyboard and monitor to spare and made sure it would boot up before reinstalling.  Installed spare server back into RF enclosure.  Restored power to Ka-SACR at 1350 GMT.  Notified mentor that server change was complete.  RDS1 that was removed will be sent to mentor at PNNL.

*Old – BSI 1027725  *New – BSI 1027728

7/3/14

BC/CAM

Mentor requested RAM in RDS1 be replaced.  New RAM was ordered and received today.  Powered RDS1 down at 1820 GMT and replaced RAM.  RTS at 1900 GMT.

7/2/14

BC/CAM

W-band antenna cover is ripped and needs to be replaced.  Removed old cover and replaced with new shrink wrap.  Secured with band and used heat gun to shrink for tight fit.  RTS at 1530 GMT.

6/11/14

BC

W-Band Drive power not reading and system RF output low.  Mentor had Prosensing send replacement parts for the RF drive assembly. Replaced all parts received and turned system on. System now reads Drive power of 8 dB and output power is 60 dB. Returned sytem to operation at 1430 GMT.

*Old Multiplier – 283 (part# AMC-10-RNH00)

*Old –

5/28/14

BC

Spectral artifacts in Ka data while dry air system running.  Inspected system at 1700GMT in anticipation of mentor troubleshooting on 29 May. Discovered solenoids in RDRAR box chattering excessively. tightened lock nut on bottom of solenoids and returned system to Mentor at 1800 GMT.

5/1/14

BC

Ka-Band Channel 2 gain low.  Instrument mentor wanted LNA changed in channel 2 of the KA RDS1. Took unit offline at 1751 GMT. Replaced *LNA with one that was previously removed from Co-Pol channel (channel 1) and returned to service at 1945 

*Old – 1038  *New – 1037   Part# LNA-28-01020

4/22/14

BC

Mentor had replacement parts shipped to site for Ka transmitter.  Replaced parts sent from Prosensing by Mentor. The parts were a 2x multiplier and mixer for Ch1 of the Ka transmitter. noticed entire system has dessicant dust everywhere and will require cleaning in the future.

4/2/14

CAM/BC

Received new air conditioner to replace failed lower unit.  Replaced air conditioner and installed a 20A receptacle to plug into rather than hard wiring.  This will allow easier removal for maintenance in the future.  Powered air conditioner on and confirmed good operation. Radars are still off line at this time at per mentor’s request.  RTS at 1430 GMT.

3/31/14

BC

KA RDS1 would not power on.  Found Ka side of SACR not powered on. Upon investigation discovered KVM switch in the bottom of the enclosure routes the control for the PDU. Removed switch and cleaned desiccant out of unit. Replaced switch in system and control was restored.

3/27/14

BC

SACR emailed stating shelter temperature was over 33º C.  Discovered both AC units blowing warm air. Removed both units and cleaned them thoroughly. Replaced ACs in System and the top unit is functioning normally. Bottom AC unit still not working

3/10/14

CAM

SACR system was completely powered down, UPS and all.   System control message on the Control Panel indicated there was a shelter over temperature (33C) and therefore a full system shut down on 2/19.  All of the coolant from above had drained back into the chiller reservoirs, which in turn overflowed.  Cleaned up mess and drained some coolant.  Powered the UPS and chillers on.  Filled chiller to just above min level.   Only one air conditioner was on at the time.  Powered second unit on and notified mentor.

1/22/14

CAM/JEM

W-SACR modulator that was installed in November has never worked due to numerous faults.  Mentor requested the original modulator be reinstalled.  Removed RF unit from W-band and lowered to ground using the telehandler.  Replaced *modulator.  Raised RF unit back to roof and installed.  Restored power to radar at 1930 GMT and notified mentor. Mentor will allow system to stabilize for a short time before attempting to transmit.  *Old – 004  *New - 013

11/14/13

CAM

Vendor sent an additional counterweight to be added to the SACR pedestal.  This is to make up for the weight that was added to the Ka radome when the new canvas cover and springs were added.  Added counterweight using the supplied longer bolts.  RTS at 1730 GMT.  Radars are not transmitting at this time.  Mentor is on site.

11/13/13

CAM

W-SACR output power has been dropping.  Mentor requested the EIKA and modulator be replaced.  Powered radar down at 1900 GMT.  Removed RF unit from W-band and lowered to ground using the telehandler.  Replaced *EIKA and **modulator.  Raised RF unit back to roof and installed.  Restored power to radar at 2230 GMT.  Mentor is on site and will let system stabilize with chiller and dehydrator running overnight before testing output.

*Old – E1122E0  *New – E1188C1

**Old – 013  **New - 004

11/13/13

CAM

Instrument mentor requested the memory in RDS1 (Ka) be swapped with the memory from the spare computer.  Replaced memory as requested at 1715 GMT.  

10/18/2013

JEM

Instrument mentor cannot access the MCC computer.  Found the computer froze up.  Rebooted the MCC computer @2030 GMT.  Notified mentor after the computer had completed boot-up.  Mentor was able to access the MCC computer.  RTS @2045 GMT.

10/7/13

CAM

Instrument mentor advised he was unable to access the MCC.  Found the MCC was locked up.  Disk drive and network LEDs were flashing, but display was blank and no response to mouse or keyboard.  Power cycled MCC at 2030 GMT.  Computer booted up and is now accessible over the network.

9/30/13

CAM/JEM

Instrument mentor advised the Ka-SACR Co-Pol LNA had failed and requested that it be replaced.   Powered down Ka-SACR at 1430 GMT.  Replaced the Ka-SACR *Co-Pol LNA.  Restored power at 1615 GMT.  Notified mentors that maintenance was complete. 

*Old – 1037  *New – 1021  Part# LNA-28-01020

Note: mentor later advised that swap did not fix problem, so SN 1037 is probably ok.

8/14/13

Vendor

Vendor notified mentors that new SACR software version 1.9 had been installed on all servers.  System was left powered up, but idle, waiting for mentors to put the radar through its paces and test the new hardware/software. 

8/5/13

CAM

The IT department has replaced the hard drive in RDS2.  Reinstalled RDS2 in RF box at 1245 GMT and powered on.  Notified mentor that servers were installed.

8/1/13

CAM

The IT department has replaced the hard drive in RDS1.  Reinstalled RDS1 in RF box at 1800 GMT.  Removed RDS2 and took to IT lab to have its hard drive replaced as well.  System remains offline at this time. 

7/8/13

CAM

Replaced the failed fan on chiller 2 with the fan from the spare chiller.  This way the radar will be ready for operation once the server hard drives are replaced.  New fans have been ordered and one will be installed back on the spare once they are received.  RTS at  1845 GMT.

6/27/13

CAM

RDS1 has been experiencing problems and locking up frequently.  The log files indicate a memory problem:

 

kernel:[  776.668391] EDAC MC0: UE row 1, channel-a= 0 channel-b= 1 labels "-": (Branch=0 DRAM-Bank=6 RDWR=Write RAS=9544 CAS=0 FATAL Err=0x1).

 

Removed RDS1 at 1900 GMT and took to computer lab for troubleshooting.

6/25/13

CAM/JEM

The fan has failed on chiller 2.  No spare available.  Manually powered the chiller off and notified mentor/vendor of the situation.  Radar is currently offline pending fan replacement.

6/25/13

CAM/JEM

The polyethylene radome on the Ka antenna is damaged and needs to be replaced.  Removed damaged radome.  Installed tensioners and the new canvas radome on the Ka antenna.  RTS status O.K. at 1930 GMT.  Radar is currently offline due to server and chiller issues.

6/24/13

CAM/JEM

Vendor sent the hardware necessary to install the Ka antenna drain, RF unit brackets, and new canvas radome.  Opened the Ka RF enclosure, disconnected all RF unit connections and slid the unit out.  Drilled and tapped a hole through the bottom of the antenna for the barbed fitting.  Cut a ¾” hole in bottom of RF enclosure for the drain valve and connected tubing between the antenna fitting and valve.  Removed stress fractured tabs on RF unit and installed new brackets as per vendors instructions.  Slid RF unit back in, secured new brackets and reconnected all lines.  Will install canvas radome on Ka antenna tomorrow.  Parked antenna at 0 degree elevation.  Radar is currently offline.

6/10/13

CAM

About ¼ of the Ka radome has pulled loose from the clamp that secures it to the antenna.  There are also several small tears in the radome.  Antenna was pointed vertical and found to have several gallons of water in it from recent rains.  There is a replacement canvas dome and hardware on site that needs to be installed.  Vendor will send hardware to install a drain in the antenna, as well as hardware to better support the RF unit.  Will perform maintenance when all parts are received.  Placed antenna at 0 degrees azimuth and 0 degrees elevation to drain water that had already accumulated in antenna and prevent further accumulation.

5/28/13

CAM

SACR pedestal is not scanning and unable to login to computer (MCC) remotely.  No data files have been collected since 5/22.  Local check found MCC was powered on and disk activity LED was flashing, but there was nothing displayed on the monitor and no response to keyboard or mouse.  Power cycled the MCC at 1420 GMT.  After MCC booted up completely, system check found rds1 was down.  Power cycled rds1 and loaded sgp_sacr_kaw_ops.pax after rds1 booted up.  After a minute or so, the status of both rds1 and rds2 changed to server (software) down.  Rebooted both servers.  Rds2 booted up normally but rds1 did not come up.  Power cycled rds1 again and it then booted normally.  Loaded sgp_sacr_kaw_ops.pax and RTS at 1443 GMT.

5/16/13

CAM

No data had been collected from Ka-SACR for 10 hours.  Connected to SACR web interface and found rds1 server was pingable, but server software was down.  Attempted to both start and restart software with no change in status.  Rebooted rds1 at 1219 GMT.  Turned on high voltage after reboot and Ka band resumed transmitting at 1222 GMT.

5/15/13

CAM

SACR pedestal is not scanning and unable to login to computer (MCC) remotely.  Local check found MCC was powered on and disk activity LED was flashing, but there was nothing displayed on the monitor and no response to keyboard or mouse.  Power cycled the MCC at 1525 GMT but it would not boot up (nothing shown on monitor, even after a couple of minutes).  Power cycled again and this time it booted up successfully after several minutes of what appeared to be hard drive scans.   A message indicated it had been 182 days since the last check, so a check was forced.   After MCC booted up completely, system check found RDS1 was down.  Power cycled RDS1 and loaded sgp_sacr_kaw_ops.pax after RDS1 booted up.  SACR RTS at 1545 GMT.

5/1/13

CAM

Observed the Ka band server (rds1) was down.  Attempted to reboot server but it would not boot up.  Power cycled rds1 and it booted normally.   Ka band started transmitting at 1445 GMT.  Pedestal was stationary and pointing vertical.  Loaded sgp_sacr_kaw_ops.pax at 1515 GMT.  Radar resumed scanning.

4/30/13

CAM

No data files had been collected from the SACR since 4/25.  Unable to bring up the web interface to check computer (MCC) status remotely.  Local check found the MCC was powered up and showing the login screen but was unresponsive to keyboard or mouse input.  HD status light was showing lots of activity, flashing constantly.  MCC would not respond to Ctrl-Alt-Del.  Power cycled MCC at 1540 GMT and it booted up normally.  Radar control panel showed RDS1 server was down.  Power cycled RDS1 and it then booted up.  Loaded sgp_sacr_kaw_ops.pax and radars returned operational at 1549 GMT.

1/19/13

CAM

Chiller 1 (Ka) was leaking where the filter housing connects to the pipe fitting on the supply line.  Ka band radar and chiller were shut down at 2056 GMT.  Removed chiller 1 filter and tightened housing 1 full turn.  Replaced filter and restored chiller power.  Observed for several minutes with no leaks.  Restarted Ka band radar and RTS at 2130 GMT. 

1/19/13

CAM

Notified mentor of the following computer errors that were observed:  The first error is in an xmessage window and says:   Xsession: Unable to launch “startkde”  Xsession --- “startkde” not found; falling back to default session.   There are actually several (at least 5) of those messages.  There is also an error stating:  There was an error starting the GNOME Settings Daemon.  Some things, such as themes, sounds, or background settings may not work correctly.  The Settings Daemon restarted too many times.  GNOME will still try to restart the Settings Daemon next time you log in.

1/8/13

CAM

Observed Ka-band server was down. Notified vendor at 2118 GMT.  Vendor rebooted computer at 2230 GMT and advised we will discuss tomorrow how to proceed regarding the frequent server outages.

1/7/13

CAM

Morning status check found Ka-band server was down.  Vendor has indicated they would like to be informed before rebooting to help them diagnose the problem. Notified vendor server was down.  Vendor responded later that they had gained some useful information and had rebooted the server at 1507 GMT.

12/24/12

CAM

No data files had been collected from Ka-band since 1819 GMT on 12/22.  Logged in remotely and found the Ka-band server was down.  Rebooted server at 1320 GMT.  Ka-band began transmitting and RTS at 1335 GMT. 

12/17/12

CAM

Chiller 1 (Ka) level alarm is toggling on and off.  Checked Chiller 1 coolant level and found it was below the minimum line.  Chiller 2 (W) was also slightly low.   Filled coolant to full line in both chillers.  RTS at 1450 GMT.

12/12/12

CAM

No data files had been collected from Ka-band since 2021 GMT.  Logged in remotely and found the Ka-band server was down.  Rebooted server at 2300 GMT.  Ka-band began transmitting and RTS at 2305 GMT. 

12/11/12

CAM

No data files had been collected from Ka-band since 2030 GMT.  Logged in remotely and found the Ka-band server was down.  Rebooted server at 2223 GMT.  Ka-band began transmitting and RTS at 2229 GMT. 

12/3/12

CAM/SPD

W-band output power had been dropping at a constant rate since may and was down to 55.5 dBm.   Normal output power should be around 62 dBm.  Vendor requested the klystron tube (EIKA) be replaced.  Powered W-band radar off at 1610 GMT and replaced the *EIKA.  Notified vendor at 1745 GMT that maintenance was complete.  Vendor powered transmitter and allowed to warm for a couple of hours before transmitting.  Received notification later that the output power had increased to 61.5 dBm and antenna power was up to 57.7 dBm. 

*Old - E1178M0  *New - E1122E0

11/20/12

CAM

Local electrical service was shut off at 1410 GMT so that Kay Electric could upgrade its power lines from 7.2 KV to 14.4 KV in Grant County.  SACR was gracefully shut down at 1302 GMT in anticipation of power outage.  Power was restored at 2115 GMT after electrical upgrades were complete.  Restarted SACR and RTS at 2233 GMT.

11/19/12

CAM

Remote status check found W-band was transmitting but Ka-band had stopped.  Digital interface debug indicated a dehydrator power fail alarm.  Accessed shelter and found dehydrator was not powered up.  Cycled the power switch with no change in status.  Reseated AC power cord and dehydrator powered up.  Restarted Ka-band radar and RTS at 2227 GMT.  Suspect the cord was knocked partially lose during previous maintenance.

11/19/12

CAM

The Ka-band chiller fan was determined to be faulty last week.   Mentor sent a spare chiller to use parts from.   Removed the fan from the spare chiller and installed on Ka-band chiller.  Fan works well but chiller is still faulting with high compressor temp.  Disabled chiller 1 monitoring in the maintenance panel to disregard faults and tested chiller operation while transmitting.  Chiller maintained set temperature of 15.1C.   Tested compressor thermistor switch (TS1) and determined it had failed (constant open).  R/R thermistor switch with one from the spare chiller.   Tested chiller operation to be ok.  Restarted Ka-band radar and RTS at 2127 GMT.

11/13/12

CAM

Site power was partially shut down at 1400 GMT in order to replace the 7 transformers that service the buildings and all instruments south of the main compound.   SACR was gracefully shut down at 1315 GMT in anticipation of maintenance.   Power was restored at approximately 1930 GMT after all transformers were replaced.  Restarted SACR and RTS at 2022 GMT.  Ka-band chiller fan was not running.  Checked fan connections and fan began running when wires were moved.  Vendor was previously aware of chiller problem and has already shipped a replacement.  Will monitor fan status and possibly replace just the fan when replacement arrives.

10/15/12

CAM

DSView page indicated no data had been received from W-band since 0538 GMT on 10/14.  Checked status and found W-band transmitter was ready, but not transmitting.  Turned HV on and transmission resumed at 1249 GMT.

10/10/12

CAM

Received an adapter from ProSensing for the serial card in the Ka-band computer (RDS1).   The adapter will allow the pedestal to communicate with the recently upgraded card.   Noted that RDS1 status indicated “server down” prior to maintenance activity.  Placed pedestal in zenith position at 1600 GMT.  Installed the adapter on RDS1 serial card and connected pedestal serial cable.  Reloaded sgp_sacr_kaw_ops.pax at 1605 GMT.  Notified Seth at ProSensing that maintenance was complete.  Seth restarted Ka-band server and advised that pedestal data was now being received.

10/9/12

CAM

No data was being collected from SACR W-band.  Logged in remotely and found W-band modulator had a Sum fault.  Reset fault and cycled modulator power.  RTS status O.K. at 1245 GMT.

10/3/12

CAM

Received a new computer (RDS1) from ProSensing for the Ka-band radar.  Stopped the Ka-band radar and powered down at 1824 GMT.  R/R *RDS1 computer.  Restored power at 1858 GMT.  New RDS1 computer was recognized by the system and could be pinged, but radar software would not start and the status was reported as “server down”.  Contacted ProSensing (James Triplet) who determined the problem was with the serial card that receives the pedestal status.  The card in the new server is an upgraded model that requires an adapter.  ProSensing will ship the necessary adapter.  In the meantime, Ka-band will be down but W-band will continue to run.  RTS at 2000 GMT.  *Old – BSI1027732  *New – BSI1027725

10/3/12

CAM

No data files had been collected from Ka-band since 2132 GMT on 10/2.  Logged in remotely and found the Ka-band server was down.  Rebooted server at 1215 GMT.  Ka-band began transmitting and RTS at 1220 GMT.  The server will be replaced later today.

10/2/12

CAM

W-band server was down.  Rebooted server at 1340 GMT.   Loaded sgp_sacr_kaw_ops.pax and W-band began transmitting at 1356 GMT.

10/1/12

CAM

No data files had been collected from Ka-band since 1720 GMT.  Logged in remotely and found the Ka-band server was down.  Restarted server at 1949 GMT.  Loaded sgp_sacr_kaw_ops.pax and Ka-band began transmitting at 2001 GMT.

10/1/12

CAM

Ka-band was not transmitting.  Loaded sgp_sacr_kaw_ops.pax at 1250 GMT.  Ka band began transmitting.  RTS at 1251 GMT.

9/20/12

CAM

No data files had been collected since 0211 GMT.  Radar was not scanning and control panel showed RDS1 server was down.  Attempted to reboot RDS1 from control panel at 1600 GMT with no success.  Cycled RDS1 power at 1611 GMT.  RDS1 server came back up and Ka-band began transmitting at 1615 GMT.  Loaded sgp_sacr_kaw_ops.pax and RTS scanning at 1618 GMT.

9/19/12

CAM

RDS1 server was not powered on.  Cycled RDS1 power remotely and loaded sgp_sacr_kaw_ops.pax PACSI.  Ka band started transmitting and RTS status O.K. at 2120 GMT.  Suspect a brief power outage earlier today may have caused the problem.

9/11/12

Mentor

Instrument mentor remotely cycled transmitter/modulator power.  Radar restarted successfully. 

9/11/12

CAM

No data files have been collected since 1809 GMT on 9/10.  Radar is not operational.  W-band shows transmitter and modulator Sum fault.  Attempted to reset fault with no change in status.  Loaded sgp_sacr_kaw_ops.pax PACSI at 1240 GMT and pedestal started scanning.  Notified mentor.

9/10/12

CAM

Jim Mead requested a measurement be made inside the Ka-band RF unit.   Stopped and disabled the pedestal at 1624 GMT.  Opened the access panel and made the requested measurement.  Resumed radar operations at 1645 GMT.  Note: W-band transmitter was faulted prior to and after maintenance.  Modulator sum fault is indicated.  Attempts to reset the fault had no effect.

8/27/12

Mentor

Found Ka-SACR not operating, restarted server and operation returned to normal at 8.27.12 0356 UTC.

8/22/12

Vendor

Brian Simakauskas advised the IDL software had been updated recently and apparently had a bug.  The problem has now been taken care of.

8/22/12

CAM

No data had been collected from the SACR since 1941 GMT on 8/21 and the pedestal was not scanning.  Opened the SACR web interface and found the radars were not transmitting.  Opened the Log Viewer and observed the following IDL error:  “IDLRTMAIN: The following error was encountered: Variable is undefined: SWPLOT.  Please consult the supplier of the application.  Execution halted at: IDLRTMAIN.”  Contacted Brian at ProSensing, who advised he would notify the programmer who wrote the code.  Loaded the PACSI file and radars resumed transmitting at 1321 GMT.

8/15/12

Vendor/

Mentor

Mentor and vendor suspect possibly the radar scans have been too low, resulting in blown LNAs due to strong ground returns.  They will modify the pasci files to not scan below 5 degrees and see if LNA failure rate improves.  

8/15/12

CAM

Instrument vendor advised the W-band Co-Pol LNA had been weak for a month or so and requested that it be replaced.   Stopped both radars and powered down at 1340 GMT.  Replaced the W-band *Co-Pol LNA.  Restored power at 1615 GMT.  Notified vendor and mentors that maintenance was complete.  *Old – 205  *New – 215

7/30/12

CAM

Instrument vendor advised they were having some problems with the Ka rds1 computer and asked that the memory modules be reseated.  Removed rds1 computer from the RF box and reseated memory.  Placed computer back in RF box and notified vendor at 1815 GMT.

7/26/12

CAM

Instrument vendor sent a “super flexible” network cable to replace the existing cable inside the pedestal between the fiber optic converter and Ka RF box bulkhead.   Powered system down at 1320 GMT.   Replaced network cable, which required disconnecting and sliding the Ka RF unit out in order to access the bulkhead fitting.  Restored power after cable replacement and RTS at 1618 GMT.

6/14/12

CAM

Instrument vendor sent replacement LNA for W-band radar and replacement hard drive for Ka-band server and asked that they be installed.  Shut radars down at 1400 GMT.  Removed Ka-band server and replaced *hard drive.  Removed all connections from W-band RF unit and modulator and slid RF unit out to allow access to RF plate.  Removed and replaced **Co-Pol LNA.  Slid RF unit back in to place and restored all connections.  Replaced all covers and notified vendor that Radars were ready for operation at 1630 GMT.  Vendor notified Site Ops later that the drive booted ok and LNA looked good.

*Old – 9VP5QZ6C (Seagate) 

*New – WMAW31052623 (Western Digital) 

**Old – 208  **New – 205  (part # LNA-10-03100)

6/4/12

CAM

ProSensing engineer advised there was a problem with the digital card in the spare Ka-band computer that was installed last week.  Removed Ka-band computer from RF enclosure at 1230 GMT.  Removed the digital card and replaced with the card from the original computer.  Connected computer to network and powered up in the shelter to allow ProSensing to test the new card.  After successful testing, reinstalled the computer in the RF enclosure at 1900 GMT.  Notified ProSensing that system is now ready for operation.

5/31/12

CAM

ProSensing requested the spare Ka-band computer be installed in the radar.  Removed original computer from Ka-band radar.  Removed PCI serial card, hard drive caddy, SATA cable, and 4 sticks (16GB) of memory from the original computer and installed in the spare.  Powered on spare computer in the building and confirmed successful boot-up.  Installed spare computer in the Ka-band radar and notified ProSensing that radar was now ready for operation.  RTS at 2045 GMT.

5/30/12

 

CAM

ProSensing engineer requested Site Ops place the spare Ka-band computer on the network so they could configure the hard drive.  The computer will then be swapped into the Ka-band radar due to failing hard drive in original computer.  Connected spare computer to network and powered on.  Computer would not boot properly and ProSensing was unable to access the hard drive containing the operating system.  T/S found failed connectors on the swappable drive caddy and SATA cable.  Temporarily connected hard drive with a spare SATA cable and ProSensing can now access the drive and will configure it for operation.

3/29/2012

CAM

Continued troubleshooting efforts to isolate cause of short circuit that is blowing fuses and backfeeding voltage to the main PDU. Vendor suspects possible slip ring failure.  Disconnected power from both Radar PDUs and powered up Radar1 and Radar2. No fuses blew. Connected only the W-band PDU (with all of its outputs disconnected) and powered on Radar1 and Radar2. All fuses blew. Disconnected the W-band PDU and connected only the Ka-band PDU (with all of its outputs disconnected). Replaced fuses and powered on Radar1 and Radar2. Only Radar1 fuses blew, but voltage was still being back fed to both legs of Radar1 240V supply. Powered Radar1 and Radar2 off and removed all fuses. Notified vendor of results and troubleshooting will continue.

3/27/2012

CAM

Installed vendor supplied desiccant packs in Ka and W band RF enclosures.  Checked rotary joint drains.  Elevation joint still appears to be leaking coolant.   

3/27/12

CAM

Instrument vendor is unable to remotely access radars through the computer.  Requested Site Ops check the network cable and converter in the pedestal.   Checked network cable and saw no visible damage, but converter showed no network link.  Neither radar system appeared to have power as all components were powered off.  Pedestal had power and could be controlled through the control panel.  Checked fuses and measured ~120V on the output of all 12.  Further troubleshooting revealed 6 fuses (3 for Ka band, 3 for W band) were actually blown but AC voltage was somehow being back-fed to the fuse panel, indicating a short somewhere between the PDU in the shelter and radar PDU’s.  The fuses that had blown were the same 6 that were just replaced the previous day.  Troubleshooting in progress. 

3/26/12

CAM

Instrument vendor notified Site Ops communications between the computer and radars were lost sometime Sunday night.  Suspect blown fuses in shelter PDU.  Checked fuses and found 3 were blown on one leg of the 240V for the Ka band as well as the 3 fuses on one leg of 240V for the W band.  Replaced fuses at 2230 GMT. 

3/15/12

CAM

Chiller coolant levels are close to red add lines on both of the reservoirs.  Filled both coolant reservoirs at 1600 GMT.  Suspect slight leak in rotary joint.  Notified vendor.

3/15/12

CM/PD

Mentor (Nitin) suspected radar PDU fuses may be blown.  Powered PDU off at 1545 GMT and checked fuses.  All fuses were good.  Restored power and notified mentor via voicemail.

2/15/12

PAD

Chiller coolant levels are close to red add lines on both of the reservoirs.  Filled both coolant reservoirs at 1445 GMT.

12/21/11

CAM/JEM

Vendor requested that the desiccant bags inside the RF enclosures be replaced.  Removed ka band RF enclosure lid and found that the original desiccant bag had ripped open and about 80% of the desiccant had spilled into the enclosure.  Due to the scanning motion of the radar, the loose desiccant had scattered all over inside the enclosure.  Desiccant was lodged in unused serial and USB ports on the computer and anywhere else it could get.  Used a shop vac to remove as much of the desiccant as possible.  Discussed the findings with the vendor and the decision was made to not put in another desiccant pack.  Also removed the desiccant pack from the W-band RF enclosure.  RTS at 1730 GMT.

12/21/11

CAM/JEM

For the past month, chiller 2 is often low on coolant while chiller 1 will be overfull.  Suspect either the azimuth or zenith rotary joint is leaking internally and thus allowing the coolant from the W-Band and Ka-band chillers to mix.  System was powered down at 1530 GMT in order to inspect the drain lines of the azimuth and zenith rotary joints.  Removed rotary joint access panels.  The azimuth rotary joint showed no signs of leaking as the drain lines were dry but the zenith rotary joint had clearly been leaking.  The horizontal section of the zenith drain line was full of coolant and there was coolant around the drain hole where it drains outside the pedestal.  Replaced access panels and notified vendor of results.  The plan is to replace both rotary joints eventually, but for now will need to continue to monitor coolant levels and fill/drain as necessary.  RTS at 1630 GMT.

12/8/11

CAM

The coolant from chiller 1 and chiller 2 seem to be mixing somewhere throughout the system.  For the past several days, when the coolant level is checked for each system, chiller 2 is low and chiller 1 is overfull.  When this condition is found, chiller 1 is always drained down to the full line and chiller 2 is filled up to the full line.  But the next time they are checked, the situation is the same, chiller 2 will be low and chiller 1 overfull.   Powered the radar and pedestal off at 1830 GMT to trace all lines in the pedestal to be sure they are connected properly.  No problems found.  Suspect the coolant from each system may be leaking inside the rotary joint and mixing there. Notified vendor of findings.  RTS at 2030 GMT.  Note that when the w-band cover was removed, a gust of wind blew it off the top of the building.  The upper right corner of the cover was damaged when it hit the ground.  Straightened the edges as much as possible.  Reinstalled the cover rotated 180 degrees so the damaged area is at the bottom.  The cover appears to seal well and the damage is only cosmetic.

12/5/11

CAM

Checked chiller 2 coolant level and found it to be midway between the full and add lines.  Removed filter from back of chiller and observed a small leak at a connection on the reservoir return line.  Removed chiller 2 from rack and removed top panel to access fitting.  Soaked up coolant that had leaked inside the chiller and tightened connection.  Filled coolant and returned chiller to rack.  Not sure if the leak was significant enough to account for the amount of coolant that was lost or not.  Will continue to monitor coolant level.  Also observed coolant leaking from the bottom of chiller 1 and pooling on top of chiller 2.  Attempted to remove reservoir cap to check level but excess pressure caused fluid to be forced out the cap.  Pump pressure was around 45 psi.  Retightened cap and powered pump off.  Loosened cap again but coolant immediately began leaking around the cap.  Drained a small amount of coolant to lower the level. Removed cap and made sure coolant was filled to full line.  Replaced cap and restarted pump.  Pressure dropped to around 38 psi.  RTS at 1700 GMT.

11/29/11

CAM

Brian Simakauskas of ProSensing received an automated email from the SACR indicating chiller 2 was low on coolant.  Checked coolant level and confirmed it was well below the minimum line.  Added coolant to the top of the reservoir.  RTS at 1840 GMT.

10/28/11

CAM

As part of continued troubleshooting to determine cause of blown fuses, vendor requested the Ka-band power supply be plugged back into the PDU.  Powered off Radar 1 and disabled pedestal.  Plugged power supply back in to PDU.  Restored power to radar and pedestal at 1635 GMT.

10/27/11

CAM

Ka-band fuses (3 rightmost) have blown again.  Vendor requested the power supply and modulator power cords be disconnected from the PDU and then the fuses replaced.  Disconnected modulator and power supply and replaced fuses at 1600 GMT. 

10/26/11

CAM

After several scans of the pedestal to test the slip ring, vendor requested power be restored to the Ka-band PDU as long as the fuses are not blown.  Checked fuses to be ok.  Inspected RF enclosure for moisture and found none.  Restored PDU power at 2145 GMT.

10/24/11

CAM

Vendor requested the Ka-band PDU AC power input be disconnected to see if fuses still blow when radar is not powered.  If so, there could be a slip ring problem.  Unplugged Ka PDU inside the RF enclosure.  Replaced the 3 rightmost fuses on distribution panel inside shelter.  RTS at 1430 GMT.  Notified vendor.

10/3/11

CAM

Vendor suspects moisture inside Radar 1 (Ka) RF housing based on high humidity readings.  Removed Ka-band RF enclosure covers at 1615 GMT for inspection and found significant condensation dripping from cold plates and chiller hoses.  Condensation is getting on and inside many components, including the PDU and modulator.  Also removed W-band RF covers for inspection but observed no condensation in Radar 2.  Left all covers off for the day to allow Ka-band to dry out.  Replaced covers at 2030 GMT.  Vendor advised Radar will remain powered off until a solution is found to prevent condensation.

9/30/11

SPD

3 fuses for RADAR1 were open in the PDU.  R&R the 3 6A fast blow fuses.  RTS at 1950.

9/28/11

CAM

Brian Simakauskas from ProSensing notified Site Ops that Radar 1 fuses were possibly blown again.  Checked Radar 1 fuses and found all 6 were open.  Replaced all 6 Radar 1 fuses with 6A very fast blow as per Brian’s request.  RTS at 1230 GMT.

8/25/11

SPD

Radar 1 had three fuses open in the PDU.  Mentor wanted the fuses changed from 3A to 5A. He also had me change all 6 fuses for radar 2.

8/19/11

SPD

Mentor Suspected the there were some fuses that were not working.  Replaced three fuses in the shelter PDU at 2044 GMT.

8/19/11

SPD

Informed to change PUMP switch on the chiller and a circuit card in the PDU.  R&R circuit card "PDU_Monitor Board" PN# 092014-0002 and crystal cpr033. No S/N for either item. Also R&R Pump switch on chiller PN#TA45-A324FJ20C0. Also R&R 2 MOV's on the power input of the chiller.

7/18/11

CAM

Instrument vendor advised that chiller #1 (Ka) is not operational and requested Site Ops contact K-O Concepts to help troubleshoot the problem.  T/S found the chiller pump or controller has failed.  K-O Concepts issued an RMA and requested the chiller be returned for warranty repair.  Removed chiller #1 to shipping/receiving.  ProSensing will send us the original shipping container and then the chiller will be sent to K-O for repair.  Switched coolant lines and data cable over to chiller #2 so the SACR can still operate.

*Old - 10080331

7/12/11

CAM

Instrument vendor notified Site Ops the chiller was not working.  Found the pump power switch was turned off.  Turned pump power switch on at 1812 GMT.