Background

A stealth mode startup was inspired to invent a new model of digital engagement that links artificial intelligence, machine learning, computer vision, and hardware to achieve a real-time realistic avatar. This solution could enable enterprise organizations to interact with people in unprecedented ways for scenarios including training and role-playing.

Facial capture that includes emotional context had been developed in Hollywood – the innovation from this startup is that the facial capture would happen in real-time. While the opportunities for film and television productions are easy to see, this client also saw a massive unmet need in the public/private domain without all the post-processing work that Hollywood spends months to create realistic avatars and scenes

Challenge

To render real-time facial features for the avatar, the solution needs to operate faster than real-time, but with technology this new, no one had experience building it.

There was no clear and obvious path for the technology, so the hardware and software elements needed to be designed together. Further, the final solution needed to be extremely efficient in terms of computation cycles without sacrificing quality and reliability.

This project included ingredients like intense time pressure, expertise in cutting-edge hardware and software, creating a seemingly impossible user experience that hadn’t been done before – all amidst an unparalleled job market with extraordinary competition for data scientists and engineers.

Kopius* was engaged because the leadership team of the digital transformation startup had heard of Kopius’ reputation for technical innovation.

Kopius was experienced working in uncertainty and could manage a project that demanded rapid prototyping and pristine technical delivery in an agile development environment.

“Kopius demonstrated ongoing agility in the face of several unknowns and variables. It was great to work with a technology partner that didn’t require us to tell them exactly what to do and when to do it so we could be successful.”

Solution

Throughout the process, Kopius’ engineering team would present concise and well-researched scientific documents that articulate architecture and engineering strategies to be considered. Then Kopius would conduct an experiment, get feedback, and create work tickets to finish a set of requirements for the upcoming sprint.

The work started by testing a range of hardware and sensor approaches, while concurrently exploring ways to capture the surface textures and features of the human face in motion. The last hurdle was to process the data received from the cameras and sensors faster than real-time.

Ultimately the solution required that the hardware capture images at 60-frames-per-second so that they could be rendered and played back at 24-frames-per-second, which is the standard for video conferencing and is perceived as seamless by the human eye. The data is stored and processed on a local workstation appliance because latency and delay hazards with cloud computing introduced too much risk to performance.

Kopius researched dozens of cameras and sensors before selecting Intel RealSense 3D cameras with an integrated RBG sensor, offering the highest depth and performance.

“Kopius’ engineers didn’t just accept the challenge – they relished it. We needed to partner with a firm that was comfortable with the unknown, could do the necessary research, had the technical expertise to innovate, and then turn out an enterprise product – and that was Kopius.” Founder, Stealth mode Startup.

“When you think about partnering with a third-party technology consultant, the relationship between engineers is especially important. Kopius’ collaboration and communication were spot on and enabled our rapid integration.”

Results

As part of the solution, Kopius also built a custom solution that generated unlimited synthetic data of faces for machine learning. The client, therefore, did not need to purchase annotated faces from third-party vendors, saving tens of thousands of dollars in training data.

The final real-time solution powers the startup’s avatars in a fully interactive 3D environment, with details like different lighting and shadows reflecting the choice of interior and exterior environments.

In addition to creating something that has never been done before, Kopius helped the startup to effortlessly collaborate between the two teams to create an enterprise-caliber platform that was more than just a science experiment. Kopius’ work has been seamlessly integrated into the startup company’s larger solution.

Kopius set this startup to be able to ship V1 of its commercial avatar product to market much faster than if it tried to organically hire the right skill sets in its early stages.  

“Kopius is our go-to partner for high-end innovation tackling the toughest problems. Their contributions accelerated our innovation and go-to-market.”

*Work was performed under the previous business name, Valence