Instrument Log  

KAZR             C1       Lamont, OK

Date

Technician

Comments

7/20/16

BMA

Mentor informed us that IDL software was down. Restarted IDL software.

7/15/16

BMA

Mentor requested we do new OS install. Started install on 7/8/16. Install was completed but transfer files could not be loaded. On 7/11/16 transfer files were loaded but system was still not operating properly. On 7/12/16 mentor remotely copied files and returned system to operation.  RTS 7/12/15  1700 GMT.

7/6/16

DKB

Powered off the transmitter at 1836 GMT, replaced the MCC HD, and loaded new vendor SW. Unable to load site specific coms files so the original HD was reinstalled and the system restarted at 2119 GMT.

6/15/16

BMA

KAZR TWTA powered down to facilitate OS upgrade. We had an issue with the USB image. Waiting for fix from mentors. OS not installed. RTS 6/15/16  1535 GMT.

6/8/16

BMA

HVAC units on shelter were inop. Had HVACs repaired. System RTS. RTS 6/8/16  2020 GMT.

6/1/16

DKB

Found the shelter temp at 99 deg. F Both HVAC units were on but blowing warm air. Cycled the breakers and both units started blowing cold air. The shelter was cooling off fast and was down to 87 deg. F by 1620 GMT.

4/12/16

DKB

Powered off and removed the TWTA under test s/n 9801257 WD30130 (from Darwin) and installed our TWTS s/n 0700763 WD36647 back in at 1419 GMT. The system is still off line for SW updates.

4/7/16

BMA

Mentor requested we install TWTA in system for testing. TWTA sn-9801257 WD30130 installed. It has tested fine. We will leave it installed to insure it continues running without faults. TWTA sn-0700763 WD36647 removed. Shutdown: 4/7/16 1700 GMT; RTS: 4/7/16 1800 GMT

3/22/16

BMA

Mentor requested we clone the KAZR hard drives in support of the Manus KAZR rebuild. Hard drives cloned and all requested equipment sent to mentor. Shut down: 3/22/16 1400 GMT. RTS: 3/23/16 1500 GMT.

12/18/15

BA

The OS hard drive in RDS1 was suspected bad by mentors. Mentors requested I clone and install OS hard drive from Darwin KAZR.  Installed cloned hard drive. Awaiting instruction from mentors.

11/17/15

BA

The site lost power at approximately 0700 GMT during a severe thunderstorm.  Power was out for approximately 6 hours.  When power was restored, KAZR did not restart.  T/S found that RDS1 was failing to boot and showed error: Icmd queue underflow (brd=0, head=3, tail=3, int_num[1871]).  Notified mentors.  System remains offline.

9/8/15

BA

Shelter was overheating. HVAC compressors not running. Restarted HVAC compressors.

8/10/15

BA

Thermostat in KAZR read 99 F. HVAC compressors were not running.

Reset breakers on HVAC units.  Warning lights on dehydrator were on for pressure and humidity. Upon opening unit, found an air hose disconnected.  Reattached air hose. Will continue to monitor to ensure problem is corrected.

7/31/15

KT

The Silica and Clay desiccant needed to be changed out on the TWTA. At 1750 GMT removed and replaced the Silica and Clay desiccant and placed back into service. RTS at 1805 GMT.

6/10/15

CAM

Power to all ground mounted transformers is being cut in order to connect the new power panel near the AOS trailer and run power to new PGS building.  Instrument needs to be shut down.  Shut KAZR down at 1328 GMT.  Restarted once power was restored and RTS at 1545 GMT.

6/4/15

CAM

Power to all ground mounted transformers is being cut in order to upgrade the transformers at AOS trailer, IDP2 and 50 profiler pad.  Instrument needs to be shut down.  Shut KAZR down at 1303 GMT.  Restarted once power was restored and RTS at 1807 GMT.

5/29/15

CAM

Mentor remotely ran fsck on RDS1 OS drive and corrected file system errors.  KAZR RTS at 1640 GMT.

5/27/15

CAM

RDS1 would not boot on 5/15/15 and a replacement data hard drive was ordered at mentor’s request.  New drive has arrived and needs to be installed.  Installed new drive and powered on computer.  Same boot error occurred as before, “Unexpected inconsistency on /dev/sdb2.  Fsck failed with exit status 4 (failed code 4)”.  Root password required to enter maintenance shell or CTRL-D to resumed system boot.  Root password is not known.  Error appears to be with OS drive, not data drive.  Reinstalled original data drive.  Notified mentor and left as is.  Awaiting further instructions. 

5/15/15

CAM

RDS1 will not boot up due to an unexpected inconsistency on /dev/sdb2.  Error indicates that fsck died with exit status 4 (failed code 4).  Root password is required for maintenance shell or CTRL-D to resume system boot.  Sent error info to mentor who advised data hard drive was bad and needed to be replaced.  System will remain down while replacement drive is on order.

5/14/15

CAM

Instruments located in KAZR and SACR vans need to be shut down to allow contractor to install shelter grounding upgrades.  Shut down system at 1642 GMT.  Powered up system but was unable to connect to RDS1 server.  IDL monitor will not start and TWTA amplifier will not power up using program software.  Performed a full system shut down and restart with no change in status.  Message log shows socket error when trying to connect to RDS1 and connection time out or refused for TWTA.  RDS1 and all other components are powered.  Notified Mentor.  RTS at 1902 GMT.

11/14/14

CAM/PAD

The antenna dehydrator compressor failed on 9/8/14 and a replacement was ordered.  Compressor has arrived and needs to be installed.  Installed new compressor and tested to confirm good operation.  Reset dehydrator runtime hours to 0 in the software maintenance panel.  Hour meter on front of dehydrator shows 8997.3.  RTS at 1910 GMT.

11/6/14

CM/KT

The silica desiccant in the TWTA dehydrator was due to be replaced.  Replaced silica desiccant pack at 2140 GMT.

9/24/14

CAM

All site power beyond the main office building needs to be shut down briefly to facilitate installation of an electrical panel at the old 50RWP pad.  Shut the KAZR down at 1438 GMT, prior to power outage.  Restarted once power was restored and RTS at 1548 GMT.

9/8/14

CAM

A rebuild kit for the compressor in the antenna dehydrator was ordered in July and has been received.  Opened dehydrator housing to remove compressor for rebuild but found it had failed.  The rods, pistons and heads were all destroyed.  A replacement compressor was ordered and antenna dehydrator will be offline pending its arrival.

6/10/14

BC

UPS batteries need replaced.  Replaced batteries in UPS. Once system resumed operation I ran a battery calibration from the UPS control software. System passed all self tests and KAZR cleared all UPS errors. System still does not report UPS voltages in KAZR software, but that is a script error. System returned to service 1920 GMT

5/29/14

BC

UPS failed at KAZR.  UPS failed and system was shut down at 0900 GMT. Restarted system and returned to service at 1820 GMT.

3/26/14

CAM

TWTA dry air silica gel desiccant bag is due to be replaced.  Replaced silica desiccant bag at 1400 GMT.

1/14/14

CAM

Bad sectors on KAZR’s spectra storage partition are causing some data files to be corrupted, resulting in failed collections and data loss.  DQPR 3938 requests the data hard drive be replaced in RDS.  Stopped the radar and powered RDS down at 1637 GMT. Replaced Western Digital 1.0TB data *hard drive and notified mentor (Nitin) that drive was ready for formatting.  Mentor remotely formatted and set the drive up for use.  Restarted radar and RTS at 2022 GMT.  Mentor confirmed data files were being written to new drive.

*Old – WCAW35170444  *New – WCAW36850369

11/26/13 –

 

11/27/13

CAM

A repaired TWTA was returned from the vendor and needed to be installed.  Installed *TWTA at 2040 GMT on 11/26.  Mentor requested the TWTA be left powered on in LOCAL STANDBY mode to warm up and stabilize overnight.  On morning of 11/27, TWTA was found powered off.  Powered TWTA on at 1418 GMT and placed in REMOTE mode after the warm up period.  TWTA went into OPERATE mode at 1425 GMT with transmit power of 159W.  RTS status O.K.

*New - 070700763 (WD 36647, WD36646)

11/19/13

CAM

Instrument mentor requested the TWTA currently installed (18) as well as the spare (12) be returned to vendor for repairs.  SN 18 has high voltage faults and SN 12 has no output power.   Removed *TWTA from the system and delivered to S/R to be returned to vendor.  Also packed TWTA SN 12 for return.  Note the KAZR has been offline since 10/18 and will remain offline pending repair of the TWTA.

*Old  - 18 WD24370 (Tube WD54591)

11/13/13

Mentor

RF unit returned from vendor and needs to be installed.  Mentor installed unit but problems with TWTA prevent returning KAZR to operation at this time.   *New – US001  (WD79101…no sticker, but that’s the number in database)

11/8/13

Mentor

Instrument mentor on site working on radars and discovered a problem with the RF unit.  RF unit pulled at 1600 GMT and returned to vendor for repair.  *Old – US001  (WD79101…no sticker, but that’s the number in database)

10/30/13

CAM

Continue troubleshooting of TWTA faults.  Mentor requested the second spare (SN15) be installed.  Powered down at 1245 GMT and R/R *TWTA with SN15.  Powered on and selected LOCAL/OPERATE following warm up.  TWTA went in to OPERATE with no faults and helix current was 0.56mA.  Noted the RF unit Radiate light did not come on.  After about a minute or two, there was a brief smell of burning electronics followed by a faint popping sound and then TWTA faulted.  Noted the helix current increased to 0.62mA just before the fault.  Powered TWTA off and contacted support engineer (Tom Kelly) at ASE.  Powered TWTA back on as suggested.  Selected OPERATE and TWTA immediately faulted.  P.S. U.V. fault LED is lit on A4 card (Coll F.O.).  Engineer suspects a failed capacitor in the high voltage power supply and recommended returning TWTA SN15 for repair.  Removed TWTA SN15 to return to vendor.  Instrument mentor will be on site next week and will look into the issues with the other two TWTAs.  Reinstalled TWTA SN18 for mentor to work with next week.  KAZR remains offline at this time.

*Old – 12 WD24365  (Tube WD48674)

*New - 18 WD24370 (Tube WD54591)

10/24/13

CAM

Mentor advised he was seeing no transmit pulses out of the spare TWTA. Requested original TWTA reinstalled in LOCAL mode for testing.  R/R *TWTA at 1230 GMT.  Also swapped the internal GPIB interfaces so now each TWTA contains its original hardware.  Placed TWTA in LOCAL/OPERATE mode but it faulted with PS OV and PS UV LEDs lit on the COLL FO card.  Cleared the fault after a couple of hours and selected OPERATE again.  This time only the PS UV LED lit.  Cleared fault and notified mentor.  T/S continues.

*Old – 18 WD24370 (Tube WD54591)

*New – 12 WD24365  (Tube WD48674)

10/23/13

CAM

Instrument mentor modified kazr.sh for TWTA to run in LOCAL mode so he can do some testing.  Requested TWTA be placed in LOCAL/OPERATE mode.  Placed TWT in LOCAL/OPERATE mode at 2118 GMT.

10/23/13

CAM

Continued troubleshooting of communications problem between RDS1 and spare TWTA that was installed on 10/22.  Moved TWTA USB connection from RDS1 to MCC computer and was still unable to command the TWTA or even see the GPIB-USB controller.  Connected GPIB-USB controller to the original TWTA and could then see the controller from both RDS1 and MCC, but still could not command TWT.  This suggested a possible problem with the spare’s internal GPIB interface.  Installed the original TWTA’s GPIB interface in the spare TWTA and could then see the GPIB-USB controller from RDS1 and MCC.  Still could not communicate directly with the TWTA via the command line, but later determined this may have been an unrelated issue.  Placed TWTA in remote mode and enabled radar services.  RDS1 successfully placed TWTA in Operate mode with no faults at 1730 GMT with helix current reading 0.55mA.  Later checked Radar Graphs on web interface and noticed transmitted power was not reporting.  Also, main bang power is reading -44 dBm, whereas it used to be around -20 dBm.  Mentor is remotely running tests to determine if TWTA is transmitting properly.

10/22/13

CAM

TWTA dry air silica gel desiccant bag is due to be replaced.  Replaced silica desiccant bag at 2000 GMT.

10/22/13

CAM

TWTA is faulting when RDS1 attempts to place it in operate mode.  The Operate LED does flash briefly, but TWTA then faults.  TWTA status in web interface shows “Failed to open twta at /dev/ttyUSB0”.  Mentor suggested trying the spare TWTA.  Powered off at 1830 GMT and R/R *TWTA. The RF input on the spare TWTA was located lower than the original TWTA so had to use a longer SMA cable between RF unit Pre-Amp out and TWTA RF In.  Restored power and enabled radar services.  RDS1 was unable to communicate with this TWTA at all.  The same status is shown in web interface, but TWTA Operate LED never flashes as though there is absolutely no communication between the two.  Placed TWTA in local mode and selected Operate and no fault occurred.  Helix current reported 0.55mA as though it was operating, but the RF unit Radiate light did not come on.  Troubleshooting continues.

*Old - 12  WD24365  (Tube WD48674)

*New - 18 WD24370 (Tube WD54591)

10/21/13

CAM

Instrument mentor advised he had remotely rebooted RDS1 on 10/18 because it was creating one large data file, but RDS1 did not come back up after the reboot.  Found RDS1 had encountered a file system error on bootup.  Fsck had died with exit status 4 (Unexpected Inconsistency) and computer recommended running fsck manually, or typing Control-D to continue.  Mentor advised to type Control-D and see if computer would finish booting.  Typed Control-D at 1520 GMT and RDS1 finished booting up, but showed “unable to enumerate USB devise at port 2” during the process.  TWP then faulted when radar services attempted to place it in operate mode.  TWT status shows Failed to open twta at /dev/ttyUSB0”.  Tried all 4 USB ports on RDS1 with the same result.  Placed TWTA in local mode to test but unit still faulted when attempting to go to Operate mode.  Notified mentor and troubleshooting continues.

8/12/13

CAM

Received a replacement T/H sensor for the antenna. Installed *T/H sensor at 1615 GMT.  Temperature and humidity are now reading correctly.  RTS status O.K.  *New - 61112424

7/15/13

CAM

The antenna T/H sensor stopped working sometime around June 28th.  T/S found the sensor has failed.  Removed *T/H sensor and will return to vendor to be exchanged for a refurbished sensor.  Will also return the sensor that failed in 2011 (SN 60387811) for exchange to keep as a spare.  *Old - 60387821

7/11/13

CAM

The dehydrator compressor failed on 6/5.  The rebuild kit that was ordered has arrived.  Rebuilt compressor and confirmed good operation.  Installed compressor in dehydrator and replaced hoses and hose clamps.  Replaced water filter element and coalescing filter element.  Powered dehydrator on and RTS at 1504 GMT.  Hour meter reading 5563.  Dehydrator started kicking on and off at random intervals.  The antenna temperature and humidity sensors are not working at this time and the humidity reading is what determines when the dehydrator should come on.  Data plots show the sensor failed sometime around 6/29.  Unable to access roof at this time due to lightning.  T/S of sensor will continue. 

7/9/13

CAM

Received an email from Seth at ProSensing saying he was now able to query the TWTA and get a response.  Communications between computer and TWTA seem to be restored.  Placed TWTA back in remote mode at 1948 GMT.  Notified mentor and he will make the changes necessary to return to normal operating mode.  He had previously modified the code to disregard TWTA errors.  RTS at 1950 GMT.

7/8/13

CAM

Continued troubleshooting of problem with computer not being able to remotely control the TWTA that was installed last week.  Powered KAZR off at 1730 GMT and replaced the *GPIB module in the TWTA with one from a spare TWTA.  Restarted KAZR but received the same error message: Failed to Open TWTA at /dev/ttyUSB0.  Vendor will log in remotely to investigate the problem.  Left the spare module in the TWTA and restarted with the TWTA operating in local mode.  RTS at 1818 GMT.  Will install the GPIB that was removed back into the spare unit.  *Old – 105   *New – 95  (model M122X-2)

7/3/13

CAM

Mentor notified Site Ops that the KAZR has stopped transmitting sometime last night.  Checked KAZR and found the RF unit was on and radiate light was lit, but TWTA was powered off.  Powered TWTA on and placed in local mode.  Placed TWTA in operate mode following the warm up and KAZR is now transmitting in local mode.  T/S of communications issue between the TWTA and computer will continue next week.  Suspect the GPIB module is faulty and will replace with one from spares.

7/2/13

CAM/JEM

TWTA has failed and needs to be swapped out with the spare.  Shut KAZR down at 1822 GMT and R/R *TWTA.  Restored power but radar software showed error for TWTA status: Failed to Open TWTA at /dev/ttyUSB0.  Rebooted computer with no change in status.  Can manually control TWTA, but computer is not seeing it.  Confirmed GPIB-USB adapter address was set to 24.  Reseated USB cable and computer saw the TWTA briefly and was able to power it on, but shortly after that the error occurred again.  If the TWTA was powered on locally, the radar software would recognize this and show status of warmup, but eventually would display the error again about the time it should switch to operate.  Left the shelter briefly to call mentor and came back in to find the TWTA had finally switched to operate (while still in remote mode).  However the error messages are still displayed for TWTA status.  Mentor is looking into problem.  Left radar operational (with error messages) at 2055 GMT.

*Old – 070700763 (WD 36647, WD36646)

*New – 12 (WD 24365, WD 48674 (tube))

6/18/13

CAM

Mentor requested TWTA desiccant bags be replaced and put on a routine schedule.  Replaced 2 TWTA cylform desiccant bags with 1 silica gel bag and 1 cylform bag at 1300 GMT.  Silica bag will be replaced every 3 months and cylform bag annually.  Noted date of change and due date on whiteboard in KAZR building.

6/5/13

CAM

Dehydrator pump failed.  Powered dehydrator off at 1300 GMT.  Ordered rebuild kit for pump.  There is a 7-8 week lead time from the manufacturer.

2/26/13

CAM

The site lost power at approximately 0350 GMT on 2/26 due to a severe winter storm.  Power was restored at 1920 GMT.  Restarted computer and RTS at 2015 GMT..

11/20/12

CAM

Local electrical service was shut off at 1410 GMT so that Kay Electric could upgrade its power lines from 7.2 KV to 14.4 KV in Grant County.  KAZR was gracefully shut down at 1303 GMT in anticipation of power outage.  Power was restored at 2115 GMT after electrical upgrades were complete.  Restarted KAZR and RTS at 2134 GMT.

11/13/12

CAM

Site power was partially shut down at 1400 GMT in order to replace the 7 transformers that service the buildings and all instruments south of the main compound.  Power was restored at approximately 1930 GMT after all transformers were replaced.  Restarted KAZR and RTS at 2032 GMT.

4/26/12

CAM/PD

SDS showed no collections from 915 RWP, DL and KAZR since the 0800 GMT collection period.  T/S found rodents had chewed through the fiber optic cables in the junction box on the side of the KAZR building.  This line feeds the hub that handles comms for all the affected instruments.  Replaced 100’ fiber optic cables between KAZR building and comms box at IDP2.  Sealed suspected rodent entry point.  Confirmed link with all instruments at 1400 GMT and the following collections were successful. 

8/9/11

PAD

CF lost power on 8/8/11 at 2247 GMT.  Power was restored on 8/9/11 at 0211 GMT.  Notified by mentor KAZR did not restart.  Received call from vendor at 1622 GMT.  Pressed reset button on MCC computer.  Computer booted up and system is now operational.  RTS at 1637 GMT.

7/14/11

CAM

Checked antenna temp/hum status a few minutes after replacing the sensor and found the sensor was already not working again.  Returned to the KAZR later in the day and removed the cable-tie nearest the sensor that was putting a hard bend in the sensor cable.  Reseated sensor but saw no change in readings.  Reseated the bulkhead connector below the antenna and the temp/hum readings returned.  Still uncertain if there is a problem with one of the wires inside the factory cable near the sensor.  Will continue to monitor status.

7/14/11

CAM

Received a replacement antenna temp/hum sensor from the vendor to replace the faulty one.  R/R *temp/hum sensor at 1220 GMT.  RTS status O.K. with valid readings at 1225 GMT.

*Old – 60387811  *New - 60468465

7/5/11

CAM

The antenna temperature and humidity sensor has stopped working again.  Removed T/H sensor from the antenna at 1500 GMT and removed the sensor tip.  Cleaned the connector pins and sockets and reseated tip.  Placed sensor back into antenna assembly and checked readings with no change in status.  Sensor appears to have failed.  Vendor will replace sensor during next site visit.

6/15/11

CAM

The antenna temperature and humidity readings are way out of range, suggesting a problem with the sensor.  Removed the temp/hum sensor from the antenna and removed the probe tip.  There was no damage or moisture observed, but after blowing out the connections and reassembling the probe the readings returned to normal.  RTS at 1445 GMT.

1/20/11

CAM

The gateway is acting as a bottleneck when collecting spectra data and bogging the system down.  Mentor requested rds1 computer be connected directly to the network, bypassing the gateway.  Connected rds1 computer’s secondary network card directly to the network using the old MMCR network connection (198.124.98.6).  RTS at 1820 GMT.

1/19/11

PAD

Mentor is unable to remotely access KAZR computer.  Powercycled Gateway located in instrument rack at 1638 GMT.  Later notified by vendor they are still unable to access.  Powercycled Gateway again at 1827 GMT.  RTS at 1842 GMT.

1/4/2011

Mentor

Instrument mentor powered MMCR down at approximately 1430 GMT and started process of replacing with KAZR.   The new KAZR will reuse the antenna and transmitter from the old MMCR.  All other components will be replaced.