U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Engineering Laboratory

Intelligent Systems Division

2021.10 UTAC Conference, Guardian Centers, Perry, GA

Share

 

Guardian Centers
Guardian Centers
Guardian Centers

October 4-8, 2021

Guardian Centers Responder Training Facility
Perry, GA

Download Complete Report

Summary

NIST’s Emergency Response Robots Project, led by Adam Jacoff, validated 20 standard test methods for evaluating aerial drone capabilities and remote pilot proficiency at the Unmanned Tactical Applications Conference. The event was hosted by FlyMotion at the Guardian Centers responder training facility in Perry, GA on October 4-8, 2021. The NIST team, which includes emergency responders and test facility managers, helped familiarize more than 120 participants from all over the world with our standard test methods and related training scenarios embedded with quantitative measures of performance.

These standard test methods have been developed by NIST with support from the Science and Technology Directorate of the U.S. Department of Homeland Security (DHS). They are being standardized through the ASTM International Standards Committee on Homeland Security Applications; Response Robots (E54.09).

Our process for developing, validating, and standardizing a growing suite of more than fifty test methods for ground, aerial, and aquatic systems, includes several types of events. We host requirements workshops with emergency responders, validation testing with robot manufacturers, competitions with robotics researchers, and standards committee meetings with all the above. We also conduct inter-laboratory experiments with other facilities that host our test methods for their own evaluations. For all these various communities, our standard test methods provide a tangible language of robot tasks that enable quantitative measures of robot capabilities and remote operator/pilot proficiency. They help guide innovation, measure progress, and highlight breakthroughs that everybody can understand and appreciate.

Figure 1) The NIST Emergency Response Robot team
Figure 1) NIST team included the following (left to right) Max Delo, U.S. Marshals Service (Ret.); Andy  Olesen, Canadian Emergency Response Robots Association; Alex Fraley, University of Maryland; Tony  Galladora, Montgomery County, MD Police Dept.; Ann Virts, NIST;James Lantz, Maryland State Police Dept.;  Adam Jacoff, NIST; Howie Stockhowe, Virginia Beach Fire Dept.; Parry Boogard, DHS FEMA Urban Search  and Rescue Task Forces.
wide variety of drone sizes
Figure 2) A wide variety of drone sizes and capabilities were used by attendees, which is essential to fully  validate the standard tests and scenarios.

[Top]

Site Overview

Site Overview
Figure 3) The ongoing NIST station was set up for the 120 participating pilots to familiarize themselves with  the spectrum of standard test methods with examples of related scenarios. There were 20 concurrent tests  and scenarios for everybody to try throughout the week.

This event also included tactical scenarios demonstrated live with actors as perpetrators, victims, and emergency responders. They showed how drones and other robots can be incorporated into fast-paced and chaotic emergency response operations including simulated gun shots, fires, and smoke.

After each scenario demonstration the NIST team embedded standard test apparatuses to guide drone pilots through a sequence of essential tasks resulting in a quantitative score. These embedded test apparatuses enable performance of drones and pilots to be measured, compared, and tracked over time, which was never possible in such complex scenarios.

A variety of firefighting, public safety, and search & rescue scenarios were conducted:

  • Wide area search of a plane crash site
  • Vehicle identifications
  • Hazardous tanker truck accident and fire
  • Suspicious vehicle takedown
  • Bus hostage situation
  • Collapsed structure search
  • Subway terrorist attack
  • Flooded house rescue with delivery of personal floatation devices
  • Flash flood bridge collapse with school bus and vehicle rescue

[Top]

Open Test Lanes

The Open Test Lanes evaluates 5 different flight paths to identify objects from safe altitudes in open environments. These tests are scalable for all sizes of aircraft to demonstrate positive control at all times with accurate perches. They can be performed outdoors or indoors to control lighting and weather. The smallest size lane fits on an indoor basketball or tennis court for small drones and/or novice pilots to practice safely without flying in the national airspace.

Standard tests used to defined altitudes, positions, orientations, and perches to evaluate both  Maneuvering and/or Payload Functionality
Figure 4) The standard tests use well defined altitudes, positions, orientations, and perches to evaluate both  Maneuvering and/or Payload Functionality.
Basic Maneuvering trials
Figure 5) There are two ways to conduct test trials in the most basic Open Test Lane. LEFT: Basic Maneuvering trials use only white buckets to guide alignments with to capture NO ZOOM images. RIGHT:  Payload Functionality trials add some operational workload to each bucket alignment task by capturing  MAX ZOOM images of the visual/thermal acuity targets. They also use alternating white and black buckets  to maximize exposure control.
Open test lanes with all white buckets are used to evaluate basic Maneuvering trials
Figure 6) Open test lanes with all white buckets are used to evaluate basic Maneuvering trials only, which is  the typical starting point for novice pilots. Test lanes with alternating white and black buckets are used to  evaluate operational readiness and system capabilities.
Open Test Lane scales
Figure 7) The Open Test Lane scales to fit inside a gymnasium or other auditorium for small aircraft at low  altitudes or outside for larger aircraft at higher altitudes.
Open Test Lane uses the exact same set of apparatuses for 5 different tests.
Figure 8) The Open Test Lane uses the exact same set of apparatuses for 5 different tests.

The Open Test Lanes were located near related scenarios for wide area search of a plane crash site and a vehicle identification scenario. Two were set up with 3 m (10 ft) spacing between omni bucket stands and one was set up with 6 m (20 ft) spacing to be operationally relevant for various scenarios. All were set up with alternating white and black bucket stands to evaluate remote pilot proficiency using the Payload Functionality variant of the tests rather than the basic Maneuvering variant of the tests that use all white buckets throughout. The Payload Functionality tests involve a modest workload for the remote pilot to pantilt-zoom, focus, and control exposure at each bucket for scoring after the trial. The alternating white and black buckets force maximum exposure control.

[Top]

Open Scenario: Wide Area Search
Quad Screen Trial Video Example - Wide Area Search

The Open Test Lane evaluates flight paths to identify objects from safe altitudes in open environments. Related scenarios include wide area searches such as this simulated plane crash site that used all the same omni bucket stands from the Open Test Lane. There are 20 targets overall, each with 5 increasingly small features to identify for a total of up to 100 points available for a complete trial. This enables comparison of scores for pilots and aircraft that can reliably perform the various bucket alignments and identify the smallest visual/thermal acuity features across all available acuity targets. The trial time limit is typically set to 20 minutes to remain within one battery charge and to maintain a schedule throughout the day for multiple pilots. Time limited trials also enable direct comparisons of scores for completeness and efficiency. Only scores using similar aircraft and trial times are directly comparable to evaluate pilot proficiency, but a variety of different aircraft can be used to compare overall scores and ease of use.

wide area search
Figure 10) LEFT: The wide area search was this simulated plane crash site across several acres. RIGHT: There  was also a separate vehicle identification scenario on a dirt road near trees.
Quad screen trial videos capture
Figure 11) Quad screen trial videos capture the following concurrent views. TOP LEFT: The pilot’s interface  showing the thermal identification of a hand warmer inside an black omni bucket stand to represent a  partially exposed survivor within the wreckage. BOTTOM LEFT: All operator inputs to the system. TOP  RIGHT: An overview of the scenario. BOTTOM RIGHT: A detailed view of the drone in proximity to the  objects of interest.

[Top]

Open Scenario: Vehicle Identification
Quad Screen Trial Video Example - Vehicle Identification
(Scenario layout training: find the errors in the sticker placements around the vehicle)

The Open Test Lane also leads pilots toward vehicle identification scenarios from safe hover altitudes above nearby obstacles. The designated flight path around the vehicle includes an orbit with equal radius and altitude to align with the 45-degree angled omni buckets on the roof (bucket targets A1, B1, C1, D1). The chosen orbit radius and altitude is based on the height of surrounding obstacles, the intended mission requirements, the aircraft’s zoom capabilities, etc. Any orbit can be used, but only trials with similar orbits and trial times are comparable. There are also precise perch positions and orientations on the road directly under the front and rear orbit positions. These perches evaluate landing accuracy along with the functional pan-tilt-zoom capabilities of the aircraft while landed at the chosen orbit radius. Perching demonstrates the aircraft’s capability to maintain surveillance while conserving battery. The perch targets are buckets under the vehicle that represent operationally significant underbody objects in shadow (bucket targets A5, C5). Each side of the vehicle has 5 exterior visual/thermal targets to identify from the chosen radius, altitude, and perch positions. There are 20 targets overall, each with 5 increasingly small features to identify for a total of up to 100 points available for a complete trial. All targets are identified from the designated flight path starting with the angled buckets on the roof to verify the altitude and radius, then exterior features or surrounding ground objects, and exterior window targets to identify while presumably looking for interior objects. All targets are on the exterior of the vehicle to eliminate potential variations in scoring due to interior target obstructions, window glare, tinting, etc. Interior target identifications can be considered bonus points because they are less comparable across trials due to changes in sunlight.

Open scenario for vehicle identification
Figure 12) LEFT: The Open scenario for vehicle identification includes a flight path with a chosen orbit radius,  altitude, and perch positions based on the height of surrounding obstacles, the intended mission  requirements, the aircraft’s zoom capability, etc. Any orbit can be used, but only trials with similar orbits and  trial times are comparable. RIGHT: The recipe for embedding 20 visual/thermal acuity targets.

[Top]

Obstructed Test Lanes

The Obstructed Test Lanes enable remote pilots to fly safe and repeatable flight paths to inspect objects within close proximity to obstructions. They include a comprehensive set of 5 different tests that guide remote pilots through various standoff positions, orientations, and perches at 2-3 m (6-10 ft) from objects. They can be performed outdoors or indoors to control lighting, weather, and access to the Global Positioning System (GPS).

Obstructed Test Lanes set up
Figure 13) LEFT: The Obstructed Test Lanes are typically set up in order of increasing difficulty left to right as  Perch, Wall, Ground, Alley, and Post. RIGHT: The Post test shown requires circumnavigating all the wayaround to align with high and low buckets while passing between the post and the wall as if inspecting a  vehicle parked along a 3 m (12 ft) curb near a building.
Perch, Wall, Alley and Post Tests

The Obstructed test apparatuses use “dual bucket alignments” to enable remote pilots to triangulate a safe standoff position by simultaneously aligning with a perpendicular bucket and its associated angled bucket. The dual bucket alignments form a right triangle with equal dimensions for the bucket separation and the aircraft standoff. They use pairs of 7.5 liter (2 gallon) buckets with 20 cm (8 in) diameter recessed targets inside at a 2 m (6 ft) spacing.

Alignment rings inside the perpendicular (90 degree) white buckets visually guide safe flight paths toward and away from the objects. Alignment rings inside the associated angled (45 degree) buckets indicate when the aircraft is in the designated safe and repeatable standoff position. That’s where a remote pilot can maintain position and operate their interface to identify increasingly small features on acuity targets inside the buckets. This modest operational workload includes zoom, focus, and exposure to capture images for scoring after the trial. White and black buckets require maximum control of exposure levels or thermal palettes to discern more details and score more points.

There are 5 different tests with increasing difficulty called Perch, Wall, Ground, Alley, and Post. The dual bucket alignments guide remote pilots through a series of 10 positions, orientations, and perches within both the standard test lanes and the operational scenarios embedded with scoring tasks. All tests and scenarios result in quantitative scores up to 100 points maximum to facilitate measurement, tracking, and comparison across different aircraft and/or remote pilots.

Procedure: The procedure is the same for all Obstructed tests. Each test has dual bucket alignments designated 1 – 4 that at are performed in a sequence. The sequence includes some backtracking to ensure the tasks are performed in various directions relative to the obstacles involved. The sequence of positions is 1 2 3 4 – 3 2 1 – 2 3 4 with the red underlined numbers indicating the backtracking part of the sequence. That results in 10 dual bucket alignments or 20 bucket alignments total.

Maneuvering Trials: A complete trial totals up to 100 points maximum for 20 bucket alignments. Points are scored using a single no zoom image of each bucket showing either a full alignment ring (5 points), a partial alignment ring (1 point), or no alignment ring (0 point).

Acuity Targets: All trials result in maneuvering scores. But Payload Functionality trials add an operational workload to identify acuity targets while aligned with buckets. The level of detail the system can discern should be known and set prior to conducting a Payload Functionality trial. Each acuity target has 5 increasingly small gap orientations to identify correctly. The smallest features are 1 mm (0.04 in) needed to read small text on shipping labels, for example. Each identifiably gap orientation is verbally conveyed to a Proctor during the trial, or when operating alone a single full zoom image of the acuity target can be captured to score after the trial.

Time Limits: Test trials are not intended to be races. But trial time limits can be imposed to minimize fatigue across multiple tests or to maintain a schedule. Trial time limits should be long enough for an “expert” to complete a perfect trial. Scores of incomplete trials due to expired time limits can only be compared to trials with the same elapsed time limits. Typical time limits are typically set to 5 minutes for Maneuvering trials and 10 minutes for Payload Functionality trials. Although any time limits may be used depending on the drone, the pilot, and the environment.

Faults: A fault is any contact with a test apparatus or any safety issue such as crossing a designated boundary between test lanes or the remote pilot flight line. Any fault results in an end of trial.

Metrics: A complete trial requires performing all the designated bucket alignments in order with no faults (contact with an apparatus) or safety issues (exceeding a boundary). If the trial is not complete, the metrics below do not apply. Keep practicing until complete trials are routinely achieved before applying these metrics:

  • Score (total points) – primary: Presuming a complete Maneuvering trial or Payload Functionality trial, the total points are a measure of the combined effectiveness of the aircraft system and the remote pilot to maneuver through all the positions, orientations, and perches necessary to align with all the buckets. Trial scores add up to 100 points maximum. These scores can be used to compare remote pilot proficiencies when using the same drone and interface in the same test method. They also can be compared to the score of an “expert” pilot, typically provided by the manufacturer, which is considered the 100th percentile of proficiency on a particular aircraft system.
  • Efficiency (elapsed time) – optional: Presuming a perfect score for a Maneuvering Trial or a Payload Functionality trial, the elapsed time is a measure of the combined efficiency of the aircraft system and the remote pilot. Elapsed trial times can help distinguish between perfect scores to identify more efficient techniques or approaches.

Note: The verge points of the dual bucket alignments designate the most efficient locations to score all buckets. However, stable hovers at the verge points between buckets can be difficult to enforce similarly for various drone sizes and pilot proficiencies. So each bucket can be scored individually from any desired proximity, understanding that the resulting scores and trial times may be negatively affected.

[Top]

Obstructed Scenario: Bus Hostage Situation
Quad Screen Trial Video Example - Bus Hostage Situation

The bus hostage scenario included role players as perpetrators and victims with actual law enforcement organizations from the region demonstrating simulated tactics including drones and a ground robot.

Bus Hostage Situation
Figure 14) LEFT: One drone dropped a mobile phone to establish communications while another surveilled  the situation. RIGHT: The ground robot also supported interactions prior to tactical intervention by law  enforcement officers.

We added quantitative measures of performance to this scenario using dual bucket alignments to guide remote pilots safely through a designated series of inspection tasks. This enables comparison of scores up to 100 points for pilots and aircraft that can reliably perform the various bucket alignments and identify the smallest visual/thermal acuity features across all available targets. The trial time limit was set to 20 minutes to remain within one battery charge and to maintain a schedule for multiple pilots. Time limited trials also enable direct comparisons of scores for completeness and efficiency. Only scores using similar aircraft and time times are directly comparable to evaluate pilot proficiency, but a variety of different aircraft can be used to compare overall scores and ease of use.

The flight paths included safe and repeatable standoff positions, orientations, and perches within 2 m (6 ft) of objects for aircraft flying the most efficient flight path. But bucket alignments can be achieved from any standoff for larger aircraft using more capable zoom lenses. The embedded scoring tasks are 7 liter (2 gallon) buckets with 20 cm (8 in) diameter recessed targets inside. The perpendicular (90 degree) white buckets provide visual alignments for the remote pilot to trust as safe vectors to approach and leave the object being inspected. The associated angled (45 degree) buckets enable triangulation to maintain a safe proximity from the object while operating their zoom, focus, and exposure to capture images for scoring after the trial.

Each set of tasks included 10 positions and orientations worth 10 points each distributed throughout the scenario. The pilot scores 5 points for getting aligned with each perpendicular bucket plus another 1-5 points for correctly identifying increasingly small features on the acuity target located inside the associated angled bucket. The smallest features to identify are 1 mm (0.04 in) representing small text on shipping labels, for example. The white and black pairs of buckets require maximum exposure control to capture images for scoring after the trial.

Obstructed Scenario: Bus Hostage Situation 2
Figure 15) There were two orbits to perform sequentially. The large dual bucket alignments defined positions #1-10 to assess the situation quickly during the initial assault. Then the smaller dual bucket  alignments defined positions #11-20 to identify key details and gather evidence. Scenarios include an  accurate perch position and orientation to demonstrate a persistent point of view that’s considered  operationally important. In this case the large bucket #10 required landing with a view through the door up  the stairs to the driver. Another exterior small bucket perch was set up for a suspicious package placed  under the vehicle. There was also an interior perch location to identify a package found inside on the floor  near a seat.
Obstructed Scenario: Bus Hostage Situation 2
Figure 16) LEFT TO RIGHT: Exterior alignment with bucket 9 and 9A (white) look inside the door directly above the perch position. The aircraft perched while aligned with buckets 10 and 10A (black). The view from  the aircraft while perched is a persistent view of the door and driver. Exterior perch positions are also  designated for underbody inspection tasks using the smaller buckets during a second phase.
Quad screen video of the bus hostage
Figure 17) The quad screen video of the bus hostage scenario showing the drone point of view in the perch position (upper left).

Large vehicles like buses can require interior searches as well. These relate to the Confined Test Lanes described later, but are typically part of these types of scenarios. They are intended only for scoring by aircraft that can safely fly inside the bus.

Obstructed Scenario: Bus Hostage Situation
Figure 18) LEFT: An interior dual bucket alignment (buckets 1 and 1A) just inside the entry door at the driver’s seat. The perpendicular (90 degree) bucket provides a safe vector to approach the apparatus until  the angled bucket (45 degree) is simultaneously aligned. That’s the safe 1m (3ft) standoff position at which  the pilot should maintain position knowing the drone is safe from obstacles and work the interface controls  for zoom, focus, and exposure to capture an image that can be scored after the trial. RIGHT: Other dual  bucket alignments were related to views of objects on the seats, under the seats, and near the back door on  the floor.
Obstructed Scenario: Bus Hostage Situation
Figure 19) LEFT: Other interior tasks using the small buckets in confined search spaces. CENTER: Two apparatuses require vertical alignments. RIGHT: Three apparatuses require horizontal alignments. This is the  final perch position to maintain a view of an object of interest.

[Top]

Obstructed Scenario: Vehicle Takedown
Quad Screen Trial Video Example - Vehicle Takedown

Obstructed Scenario: Vehicle Takedown
Figure 20) LEFT: Vehicles of all sizes and conditions can be bucketized similarly for quantitative scoring, including panel vans and trucks. RIGHT: Dual bucket alignments guide the drone into position to look inside  the front windshield from two different angles while using triangulation between the buckets to stay safe.
Obstructed Scenario: Vehicle Takedown
Figure 21) The bucket numbering leads the pilot around an orbit starting in front of the vehicle, then around to the driver side, and so on. The large buckets targets are identified in the first orbit to assess the situation for up to 100 points. The small buckets targets are identified in the second orbit as evidence gathering for  another 100 points, including where the gun landed during the demonstration.
Obstructed Scenario: Vehicle Takedown
Figure 22) The quad screen video of the tactical vehicle takedown with evidence gathering.

[Top]

Obstructed Scenario: Hazardous Tanker Truck Accident and Fire
Quad Screen Trial Video Example - Hazardous Tanker Fire

The tanker truck accident and fire scenario used 5 large bucket apparatuses to guide remote pilots to points of view around the vehicles for a fast tempo initial assessment. We also embedded 5 small bucket apparatuses near features needing detailed inspections such as the inside of the cab, dripping valves, and gauges to read. This scenario included drop accuracy tasks to emplace remote sensors trying to determine the nature of the hazard (see the sensors on the ground).

Obstructed Scenario: Hazardous Tanker Truck Accident and Fire
Figure 23) LEFT: The tanker accident scenario included a simulated fire and potential hazmat spill requiring firefighters to react as appropriate based on reconnaissance from the drones and dropped sensors. RIGHT:  Thermal overview image from the drone and the dropped sensor packages.
Obstructed Scenario: Hazardous Tanker Truck Accident and Fire
Figure 24) Our quad screen video captures simultaneous views of what the remote pilot sees through the interface, all interactions with the interface, and off-board views of the situation around the aircraft.

[Top]

Obstructed Scenario: Structure Exterior Inspection
Quad Screen Trial Video Example - Structure Exterior

Obstructed Scenario: Structure Exterior Inspection
Figure 25) The collapsed structure had several different scenarios to fly. The Obstructed scenarios (some  shown with ovals) were perimeter windows, doors, and surrounding ground objects. The Confined scenarios  (some shown with rectangles) were vehicles in the semi-collapsed parking garage and room-to-room  searches inside the semi-collapsed building.

Structure exterior inspection tasks include looking through windows and doors along with surrounding ground objects of interest. In this case, the tasks were embedded around a partially collapsed structure. The objective was to safely fly in close proximity of about 2 m (6 ft) from the windows and doors to perform a window/door clearing maneuver with high/low and left/right views inside the structure.

Obstructed Scenario: Structure Exterior Inspection
Figure 26) Obstructed apparatuses are set up in sets of vertical and horizontal tasks throughout the scenario.
Obstructed Scenario: Structure Exterior Inspection
Figure 27) The scenario façade was embedded with window inspection tasks on two sides with partially collapsed walls. Each Dual Bucket Alignment guides the remote pilot to positions and orientations with  high/low and left/right views inside the windows.
Obstructed Scenario: Structure Exterior Inspection
Figure 28) Our quad screen video captures simultaneous views of a trial, including what the remote pilot sees through the interface, all interactions with the interface, and off-board views showing the situation  around the aircraft.
Obstructed Scenario: Structure Exterior Inspection
Figure 29) This final vertical test apparatus leads the remote pilot to perch with a view through the doorway. This was the 10th and final position and orientation in the scenario that totaled up to 100 points for a  complete search through the windows of the façade on two sides of the structure.

[Top]

Obstructed Scenario: Flooded House and Collapsed Bridge
Quad Screen Trial Video Example - Flooded House Scenario

Several houses were submerged with victims on the roof and others in the water nearby. The tasks to perform included delivery of personal floatation devices to those stranded on the roof, searching in windows for survivors, and identifying/tracking moving swimmers. The omni bucket apparatuses replaced the actors as objects to identify and the landing served as target for the dropped payload accuracy.

Obstructed Scenario: Flooded House and Collapsed Bridge
Figure 30) Victims on the rooftop of a flooded house were replaced with omni bucket stands to perform identification tasks and a landing was used as an accuracy target for dropping a personal floatation device.
Obstructed Scenario: Flooded House and Collapsed Bridge
Figure 31) LEFT: Other inspection tasks required the drone to hover low above the water to look in windows and doors. The two positions and orientations designated by the white and black pairs of buckets guided  the pilot to completely inspect the floating mattress looking for survivors (visual or thermal). RIGHT: Similar  window inspection tasks were located under a more difficult covered porch near a door. That’s where  underwater versions of these same tests were embedded for underwater system reconaissance.
Obstructed Scenario: Flooded House and Collapsed Bridge
Figure 32) A floating omni bucket task was pulled through the water by a remotely controlled rescue device to represent floating or swimming victims. It provided a moving object for the drone to identify on all sides.
Obstructed Scenario: Flooded House and Collapsed Bridge
Figure 33) The flashflood bridge collapse included an overturned school bus and vehicles requiring a comprehensive search for survivors.

[Top]

Obstructed Scenario: Night Operations House Surveillance and Search

This scenario was actually conducted the week prior at a different event, but nicely augments all of the scenarios above. This house surveillance at night used two sets of horizontal and vertical Obstructed test apparatuses on all four sides of the house, guiding the remote drone pilot to safe locations among a variety of very difficult obstacles such as the overhanging roof, trees, shrubs, power lines, shrubs, etc. A set of 5 vertical test apparatuses with 10 positions and orientations were attached to all sides of the house totaling 100 points. A set of and 5 horizontal test apparatuses with 10 positions and orientations were also placed on or near objects of interest around the house for another 100 points.

Obstructed Scenario: Night Operations House Surveillance and Search
Figure 34). Obstructed test apparatuses embedded into an exterior house search at night show the designated hover positions and orientations including a perch position in view of the back door. The aircraft  was outfitted with lights to illuminate the scene but also used its thermal vision capabilities to negotiate the  myriad of obstacles such as trees, power lines, shrubs, etc.

[Top]

Confined Test Lanes

The Confined Test Lanes enable remote pilots to fly safe and repeatable flight paths to inspect objects within confined environments and interior room-to-room searches. They include a comprehensive set of 5 different tests that guide remote pilots through various standoff positions, orientations, and perches at 1 m (3 ft) from objects. They are half the size of the Obstructed Test Lanes but use all the same procedures and scoring. They can be performed outdoors or indoors to control lighting, weather, and access to the Global Positioning System (GPS).

The Confined Test Lanes and related scenarios use “dual bucket alignments” to enable remote pilots to triangulate a safe standoff position by simultaneously aligning with a perpendicular bucket and its associated angled bucket. The dual bucket alignments form a right triangle with equal dimensions for the bucket separation and the aircraft standoff. They use pairs of 1 liter (1 quart) buckets with 10 cm (4 in) diameter recessed targets inside at a 1 m (3 ft) spacing.

Obstructed Scenario: Night Operations House Surveillance and Search
Figure 35) The Confined Test Lanes are half the size of the Obstructed Test Lane but use all the same  procedures and scoring. The are typically placed near related scenarios such as this semi-collapsed building  and parking decks with interior room-to-room searches and other confined space inspections. LEFT TO  RIGHT: Perch, Wall, Ground, Alley, and Post tests evaluate positions, orientations and perches within 1 m (3  ft) of objects to identify features.

Alignment rings inside the perpendicular (90 degree) white buckets visually guide remote pilots along safe flight paths toward and away from the objects being inspected. Alignment rings inside the associated angled (45 degree) buckets indicate when the aircraft is in the designated safe and repeatable standoff position. That’s where a remote pilot can maintain position and operate their interface to identify increasingly small features on acuity targets inside the buckets. This modest operational workload includes zoom, focus, and exposure to capture images for scoring after the trial. White and black bucket pairs require maximum control of exposure levels or thermal palettes to discern more details and score more points.

There are 5 different tests with increasing difficulty called Perch, Wall, Ground, Alley, and Post. The dual bucket alignments guide remote pilots through a series of 10 positions, orientations, and perches within both the standard test lanes and the operational scenarios embedded with scoring tasks. All tests and scenarios result in quantitative scores up to 100 points maximum to facilitate measurement, tracking, and comparison across different aircraft and/or remote pilots.

Scoring: Each dual bucket alignment is worth up to 10 points, including 5 points for maneuvering and 5 points for visual/thermal acuity. Maneuvering points are scored using a single no zoom image of each perpendicular bucket showing either a full alignment ring (5 points), a partial alignment ring (1 point), or no alignment ring (0 point). Visual/thermal acuity points are scored using targets inside the angled buckets with 5 increasingly small gap orientations to identify correctly (1 point each up to 5 points). The smallest features to identify are 1 mm (0.04 in) representing small text on shipping labels, for example. Target identifications can be verbally conveyed to a Proctor during the trial, or a single full zoom image of each acuity target can be used to score the trial after landing. A complete trial includes 20 bucket alignments totaling up to 100 points maximum per trial.

Note: The dual bucket alignments define the most efficient positions to score all buckets placed around the object being inspected. However, stable hovers at the verge points between buckets can be difficult to enforce similarly for various aircraft sizes and pilot proficiencies. So each bucket can be scored individually from any desired proximity, understanding that the resulting scores and trial times may be negatively affected.

[Top]

Confined Scenario: Collapsed Structure Search and Inspection

One of the confined scenarios at this event were semi-collapsed structures representing environments not safe for responders to search given the potential for further collapse due to aftershock or structural failures. The inspection tasks were placed around crushed vehicles within a collapsed structure. They were also distributed across individual rooms in a search pattern to ensure each room was inspected for victims and other objects of interest.

Confined Scenario: Collapsed Structure Search and Inspection
Figure 36) The collapsed structure scenario at the Guardian Centers supported a wide variety of embedded  test methods inside the structure and around objects of interest in semi-collapsed areas.
Confined Scenario: Collapsed Structure Search and Inspection
Figure 37) LEFT: Confined test apparatuses embedded into the semi-collapsed parking garage around vehicles that needed to be searched for survivors within a very complex three-dimensional environment.  We placed 5 apparatuses around the vehicles to designate 10 positions and orientations to inspect. The  white and black bucket pairs total up to 100 points for a complete trial. RIGHT: Confined test apparatuses  embedded in a room-to-room search within the collapsed structure. These measured the completeness of  the room-to-room search and quantify the detailed inspections of key features within the rooms. This  scenario also totaled 100 points for a complete trial.

[Top]

Confined Scenario: Subway Terrorist Attack

The subway terrorist attack scenario was in a simulated tunnel with multiple DC Metro railcars. A line of intact and upright railcars near the station entrance led to one railcar on its side almost blocking the tunnel. The scenario included several survivors inside and around the perimeter of the overturned railcar along with some suspicious packages that could have been secondary explosives.

Confined Scenario: Subway Terrorist Attack
Figure 38) The long dark tunnel started with an intact set of railcars near at the simulated station entrance.
Confined Scenario: Subway Terrorist Attack 2
Figure 39) LEFT: We used Confined dual bucket alignments to designate 10 positions and orientations outside the subway car where victims were located along with suspicious packages. RIGHT: Two drones  flew in tandem to help one another with side views of their situations as they negotiated the complex  interior of the overturned railcar.
Quad Screen Video
Figure 40) The quad screen video shows the first dual bucket alignment in the sideways entrance door. This was the first of 10 positions and orientations inside the subway car to measure how far in a drone could find  victims and identify suspicious packages. 
Drones were assisted by two four-legged robots
Figure 41) LEFT: The drones were assisted downrange by two four-legged robots. RIGHT: The environment was dark so thermal cameras were used extensively. This view is of the improvised pilot station on the  subway tracks, set up near the overturned subway car rather than at the start of the tunnel reduce radio  communications issues and flight times for the training scenario. 

[Top]

Legged Mobility and Search Tests

Newly developed standard terrain tests for robots with legged mobility are based on plastic crates. They are quick and easy to reconfigure into increasingly difficult terrains. For example, simply as a flat terrain they have a grate-like surface that some autonomous robots (and animals) don’t see so well. Various terrains can be reconfigured including the Diagonal Hill (shown) or a Pyramid Hill with the highest point in the center of the room. Or every second crate can be flipped over in a checkered pattern to include negative obstacles or holes to avoid, forcing very specific footfall locations. These checkered terrains can be combined with the hill topologies for maximum difficulty. These crate-based terrain tests are also easy to purchase and fabricate all over the world. Note the white and black dual bucket alignment tasks all around the sides and top at different elevations and orientations. This is a quantitative way to measure and compare the search/inspect capabilities of robots while negotiating the terrains. The same search/inspect tasks can be placed over all the other standard terrains such as crossing ramps, sand and gravel crossover slopes, and stepfields, etc.

Zig-zag crate terrain
Figure 42) LEFT: The zig-zag crate terrain is easy to reconfigure for increasingly difficult challenges. CENTER: The diagonal hill with no holes. RIGHT: The checkered version of the terrain with holes either to avoid or  deal with.
Boston Dynamics Spot
Figure 43) The Boston Dynamics Spot without its manipulator or extra sensors practicing the crate terrains.
Legged Mobility and Search Tests 3
Figure 44) LEFT: Embedded search/inspect tasks with dual bucket alignments that measure the difference between robots with and without manipulators that can position, orient, and zoom cameras while  negotiating various terrains.  The search/inspect tasks are exactly like the aerial Confined Test Lane tasks  but require a closer 30 cm (12 in) standoff position to align with each pair of buckets simultaneously. RIGHT:  Fetch tasks may be the next set of tests to standardize!

[Top]

Sensor Test Lane

The Sensor Test Lane evaluates visual acuity, thermal acuity, hazmat label identification, motion detection, and latency while hovering at altitudes up to 90 m (300 ft). Also Survey Acuity for forward flying aircraft. It uses larger 120 liter (32 gallon) buckets with 45 cm (18 in) diameter inscribed rings to guide alignments. Panels with targets on white and black backgrounds point toward the hover position. Typical visual/thermal acuity targets are across the top line and operationally relevant objects are across the bottom line, including hazmat labels, partial license plates, gauges to read, weapons to identify, and thermal sources as examples. The back side of the target panels have QR codes of different sizes to capture in flyover video to measure the Survey Acuity of the system from various straight and level flight altitudes. The video can be postprocessed to automatically read as many incrementally small QR codes as possible for scoring. This can identify the smallest readable size across a large search area when multiple posters are laid out.

Sensor Test Lane
Figure 45) LEFT: The Sensor Test Lane was set up on the centerline of an existing street. This makes it easy  to ensure a straight line with measured distances between buckets and targets. The street also makes it  easier to pre-program flight path using existing satellite imagery to fly straight and level over the centerline.  RIGHT: The typical visual/thermal acuity targets on white and black backgrounds point toward the hover  positions for the pilot to read. The back side of the A-frame panels have various size QR codes to read as a Survey Acuity test for forward flying aircraft.

  [Top]

Contacts

Created February 16, 2022, Updated February 25, 2022