Combining drone and 360 camera video stream for independent applications

(P19123: Amelia Drone Project)

By Bryan Bausinger, John Cowan, Noah George, Jacob Kenin, Zachary Stewart

Mechanical, Computer, and Electrical Engineering in Multi-Disciplinary Senior Design at Rochester Institute of Technology. 1 Lomb Memorial Drive, Rochester, NY. Email: bab1943@rit.edu

ABSTRACT

The current drone market caters to video playback, or recreation. The goal of this project is to research and implement a strategy to transmit a 360 video from a drone platform to a base station at real time. The drone is built to lift a harness packed with transmission and video hardware that sends its data down to the base station. The base station employs software to convert the transmitted data into standard video formats which may be injected into a virtual reality headset. Given a clear line of sight within a ¼ mile range, the video stream is maintained at 1080p at 8 fps with an alternative 720p with 30 fps. The limitation of this is the software as there exists enough throughput in the design for 2K resolution at 30 fps.

This is the first pass at prototyping a potential solution. The drone platform is stable and meets the basic requirements. Future research may explore maximizing resolution, improving interfaces, and improving cabling. A team taking over the project is recommended to consist of the following engineers: Mechanical (payload/drone hardware), Electrical (drone electronics and cables), Software (camera data out), Computer (data transmission), Software (UI and misc.), and Electrical/Mechanical (Drone expert and team leader).

KEYWORDS: RIT, MSD, Amelia Drone, Lockheed Martin, Research, 360, sphere, hemisphere, camera, transmission, throughput, VR, live

BACKGROUND/MOTIVATION

Current drones in the market serve one of two primary functions, cinematography for picture and video, or recreation. There are other niches being explored such as transportation of goods for shipping and farming. We hope to break into a new application of surveillance

and inspection with the use of a spherical camera. Scenarios for use include the flight through urban environments for safe search of troops and terrain, inspection of hard to reach hardware, and many others.

The problem consists of connecting a 360 spherical camera to an existing drone platform. The video is then streamed live to a base station containing a pilot operating a virtual reality (VR) headset. The video must be streamed live and recorded for later playback.

Lockheed Martin sponsored this project. They desire the demonstration of the throughput of data from a moving platform to the base. The pilot of the drone is considered to have experience but the visual in the headset must be stable for the pilot’s comfort. An overall safety consideration for nearby wildlife and personnel requires the consistency of the drone. Areas not included in this iteration, but considered were Light Detection and Ranging (LIDAR) for obstacle detection during drone flight, and retractable legs on the drone for a larger image in frame.

The goals specified by Lockheed resolved into the importance of the data transfer. The drone platform is considered well explored as an area of research. Ours must provide the most basic functionality and flight times. The throughput of this mass amount of data is the deliverable Lockheed is focused on. Directionality and range on a moving platform are completely new ventures. The prototype must deliver a smooth visual with a high quality in real time.

From Lockheed’s requests, Engineering Requirements (ER) were constructed. We verified that in developing these, each customer requirement was covered by one or more of the ERs. Each requirement also must be quantifiable and testable. The correlation between customer requirements and ERs is shown in figure 1.

Items that were not testable but important factors in design were considered constraints in figure 2.

Figure 1: Requirement Flowdown

Figure 2: Constraints

The anticipated result of this project is a functional prototype in an open field with no obstacles between the base and drone. The major deliverable is the video, both live and playback, to show the viability of radio signals at high throughput.

DESCRIPTION OF DESIGN

We separated the project into separate subsystems and specified the necessary connection between them. The subsystems consisted of Drone, Base Station, Transmission, Camera, Software, and Drone Payload Frame.

The driving consideration in selecting the camera is the ability to livestream. The criteria evaluated are the output video format, developer support, and attachment method to drone. The Ricoh Theta V provides micro-usb output, full support, and ¼-20” screw mount.

Ricoh Theta V stock livestream requires throughput of 120Mbps at 4K and 42Mbps at 2K. Although our resolution is lower than these, our transmission selection revolved around directionality and throughput. With the support of Subject Matter Expert Professor Indovina, Ubiquiti point to point transmitters were selected. The transmitters peak at a throughput of 280 Mbps.

Figure 4: Subsystem Requirement Flowdown

Figure 3: Ubiquiti Rocket Horizontal Directionality

The Base Station consists of the processing (computer) and VR headset. The VR headsets all function very similarly so the criteria were primarily cost and necessary inputs. From this, the Oculus Rift was selected. The computer must provide output to the Oculus (3 USB 2/3 ports, 1 HDMI port) and receive input from the transmitter (1 Ethernet). Also, the computer must support the drivers determined to be at least 8 GB RAM, 1050 TI GPU, and Intel i3-6100 CPU. The Acer 5 Nitro met all parameters at the lowest cost.

The other factor of the drone is the ability to support the weight. A budget was created in order to verify our adherence to this consideration. The two platforms evaluated were the DJI S900 and the Guai 950H. The DJI S900 was initially selected because of the simplicity of assembly. However, sourcing complications forced the group to select the Guai 950H. This platform provides a maximum take-off mass of 11,930g.

Figure 5: Drone Payload Weights

The Drone Payload Frame must house the camera, transmission components, and attach to the drone. In order to minimize weight, decrease assembly time, and increase customization, additive manufacturing was the process selected. Iterative design first began with

roughly form fitting all components. Discoveries about wire management and space led to a second iteration. Upon testing of the camera, a hardware solution for stabilization became necessary. This is included through the use of the DJI Osmo Mobile 2.

The software development of the Amelia project was challenging and had moments of one step forward, two steps back. Throughout the different attempts at designing a program, we found it was very simple to implement in some areas but very challenging in another. Towards the end of the time of the project, there was a realization that the software on the camera itself gave limitations and with time and understanding of android app development, this restriction can be removed.

After attempting to use unity and losing a time in the second semester, the development of the electron browser based application was our final decision on the software moving forward. This application was planned to be all in one product however it caused slight problems as well. Unlike unity where it was simple for VR, electron caused issues because the chromium version that it is based on for our project does not support any WebVR or occulus VR drivers. In order to fix this, we made it that the electron application hosts the code and acts as a local host for a firefox browser to connect to it. In the software diagram, there are the two main HTML files for this project. The Index.html file is what is used for the firefox front end. This includes our settings selection for the ricoh theta, and also has the A-frame program we are using in it. The A-frame is an open source online program where it can render 3d objects and work with WebVR to create VR environments in web browsers. By creating a videosphere and connecting that to the incoming stream of the theta, the VR video can be displayed in the headset and browser.

In the end, the software did not reach all the goals of the project. It fell flat at the resolution and frame rate. Near the end of development, it was found that the chosen camera could only stream the MJPEG format we were using at certain restricted resolution and frame rates. When this was discovered, the only plan forward to try and fix this was to develop an

application to run on the ricoh camera itself, since it runs android os, and have that application mimic the cameras getLivePreview function. This was a challenging goal to get working as it was completely new area for the developers of creating android applications. The camera itself has been tested to record 4k 30fps stably. In the end, this is what is highly recommended to be looked into and worked on for the next set of developers.

Figure 6: Payload configuration (brown = gimbal, white = transmission, black = 3D printed, grey = camera, orange = power)

This preliminary design accomplishes all of the customer requirements, drone functionality, data throughput, and safety. However, there exist risks and reservations at the end of the design phase. The throughput of the transmitters is highly dependent on the noise of the environment. Due to the test specification being an open field with no obstacles, this is considered to be miniscule. The drone assembly is performed by a mechanical engineering team with no prior experience. The overall software package is a large undertaking with limited personnel and no prior experience. These three factors drive our first testing areas and the largest uses of our available resources.

Figure 7: Software Diagram

Current drones in the market offer the ability to record a normal image, not 360 sphere, for playback, not live. This drone pushes the envelope by testing the range of Wi-Fi transmission and offers a virtual reality experience that is separate from the flight controls.

SUPPORTING FEASIBILITY EVIDENCE

It is crucial that the Drone Payload Frame does not become unseated during operation. One way this would happen is if the bolts sheared or slipped off. Following hand calculations, this failure mode was put to rest. Another consideration largely involving the frame is the effect of center of gravity (CG) on drone performance. The mounting of both the drone battery and payload are implemented with slotted or Velcro attachment. This allows to adjust to slight margins and correct for an off-center CG. Discussions with Subject Matter Expert Dr. Crassidis validated this approach.

Figure 8: Drone center of gravity

The throughput and its effects due to noise, distance, and directionality were variables that needed validation upon arrival. The antenna and receiver were paired and experimented within the range of 1/8 mi in a high noise environment and variable directionality. This stress test showed a minimum throughput of 40 Mbps, far beyond the necessary value for a 1080p video and nearing on the requirement for a 2K video.

Research revealed that there were some considerations of the camera overheating. Through discussion with Dr. Stevens, the camera is modeled as a fin and adiabatic in the mount as shown in figure 9. Using the expected operating conditions and current mounting, the maximum ambient temperature is 87F. For future iterations, if necessary, a heat sink or different mounting condition may increase the allowable ambient conditions.

Figure 9: Camera heat considerations

Computers with similar parameters to those selected are stress tested to validate the overall functionality. A VR viewing application is ran while downloading a file at a rate of 120 Mbps and the RAM, GPU, and CPU usage are monitored. The test showed 50% usage of the CPU, 4GB usage of RAM, and smooth visual resulting from

the GPU. During operation, we expect slightly more strenuous conditions due to the processing of video. However, the capability is demonstrated and permissible to move forward.

RESULTS

The resulting prototype of the combination of all subassemblies is shown in figure 10. Combination of peripherals in the drone required specific wiring considerations shown in figure 11. For detailed instructions on drone assembly, see the link:

Figure 10: Drone configuration

Figure 11: Drone wiring

Figure 12: Drone Payload Side View

Drone flight follows standard processes that we have documented. The software package and attachments require specific orientations and setup processes shown in the user manual. The manual also contains the troubleshooting processes. All of this information may be seen via [1].

Success is validated based on the table shown in figure 13. Each of the tests has a specific documented process that is repeatable. Due to time constraints, some requirements were unable to be validated by a physical test. However, the process is laid out for the future.

Figure 13: Test Matrix

Drone functionality was a major setback in the assembly process. Due to the shift in providers, new propellers, motors, and power distribution board had to be retrofitted along with an extended calibration process. This caused a major setback in testing schedule and limited familiarity with the drone flight. Despite this, the physical drone passed the height, distance, and

stability requirements for use. During testing, new insights occurred such as altering the automated take-off procedure.

The software testing revealed difficulties in pulling the information out of the camera. Given our approach, the API implemented is limited in resolution and frame rates that are preset. Future research may be able to circumvent this limitation. As it stands the limitation on the resolution and frame rate is within the software itself, not hardware.

Drone flight time is a direct result of the payload and the battery selected. The later addition of the gimbal caused this total time aloft to decrease substantially. Areas to improve this could be including a larger battery (although there is a cost benefit of power versus weight), decreasing payload weight, or improving efficiency of motor/propellers. Each of these is a large implementation.

The frame lost during flight is likely due to the software. The transmitter testing independent from our system showed consistent throughput at various angles and distances. Therefore, further testing of the software is necessary to get to the root of this discrepancy.

CONCLUSION

The budget provided was $5,000 and after all purchases we come in just under as shown in figure 14. The major expenditures are the drone platform, base station computer, VR headset, transmitters, and camera as anticipated.

Complications with the software and drone shifted back the schedule severely. This is primarily due to the lack of expertise in the respective areas. Through discussions of the customer, we reduced requirements to validate the basis of the technology, a flying drone platform, and minimal video resolution.

In retrospect, the approach would vary. First, the team needs more computer and electrical engineers than mechanical. There would be a larger importance put on the software very early on with hard deadline requirements. The overall software is unexplored and takes more hours than anticipated. The drone build should begin earlier in the process. We did not account

for mechanical assembly, electrical assembly, programming, and troubleshooting. The 3D printed payload is extremely useful and is the optimal solution for a lightweight attachment housing all components. The computer, camera, headset, and transmitter would be selected again as they provide the parameters expected and necessary in our design.

Figure 14: Project Budget

The drone platform can be better balanced and investigated in the future for performance. Time constraint forced our team’s familiarity with flight to be limited. Smoother platform results in a smoother image.

The software provides the basis for the transmission. The camera has higher resolution available and transmitter has available throughput. Therefore, the limiting factor is software which may be worked in the future.

RECOMMENDATIONS

The theoretical throughput of the transmission system is upwards of 300 Mbps. Through testing at various ranges and directionalities, a minimal throughput of 42

Mbps is discovered. Research into the effect of bandwidth and noise may better optimize for higher throughput.

The software package utilizes approximately 7 Mbps at 1080p 30 fps. This is due to complications retrieving the data from the camera. A software or computer engineer may be able to better dissect this function and improve the resolution and frame rate.

Experience with drone usage assists in a future teams ability to improve the platform. There was little time for the drone to be tested and therefore no time to adjust any weight distribution or hardware. More flight hours will reveal more areas for improvement or optimization.

KEY LEARNINGS

The goal of this course is to prepare each student for the challenges they will face in the workplace. The students garnished new experiences and learned new lessons throughout the course. These varied from the extensive planning exercises to sufficient documentation of any design changes.

One of the first lessons the team learned was that the project was not just our own. Unlike school, the project was owned by our customer and our guide oversaw our progress. It was a new experience having two separate bosses that have a vested interest that goes beyond the end result. This project required much more communication with our guide and customers than any of us were used to but by the time the project was wrapping up, the external communication standard was established.

As we dealt with complex procedures and solutions, the way we solved problems changed. At the start of the project we were comprised of individuals doing our own separate tasks toward the goal; but as the project progressed, as did the problems. Complex problems require complex solutions which often require the assistance of multiple people. With the co-investment into the solution, the way of documenting and solving problems changed as many other parts relied on each other.

Finally, the team learned about the importance of a schedule. At times the deadlines seemed very arbitrary

since there was a deceptive surplus of time but we quickly realized that the importance of those deadlines. Without setting limits and action items to the deadlines we would have wasted an immense amount of time on solutions that would not have worked. Once we quantified the amount of time that each potential solution was worth we could pivot into a new solution with better potential which often worked better than the original idea.

ACKNOWLEDGEMENTS

Thank you to Lockheed Martin and Jim Melby for sponsoring the project as well as being instrumental in guiding us through the process. Thank you to Gary Werth and the RIT MSD team for providing the guidance in all aspects from planning to scheduling to developing these technologies.

The subject matter experts at RIT were instrumental in coming to some of the optimal solutions. These professors include Dr. Indovina (data transmission and signal), Dr. Stevens (heat transfer), and Dr. Crassidis (drone).

REFERENCES

[1] Bausinger, Bryan, et al. “P19123: Lockheed Amelia Drone.” EDGE, RIT MSD, edge.rit.edu/edge/P19123/public/Home.