1. Home
  2. Google Slide
  3. TheHeretic MakingOf DigitalDragons

TheHeretic MakingOf DigitalDragons - 🟧Sourceful

Making of a real-time short film: digital humans and vfx characters - 🟧Sourceful

animation, VFX, films, animation technology

The Heretic

Making of a real-time short film: digital humans and vfx characters

Welcome to the Making of The Heretic - a real-time short film made with Unity.


Unity Demo Team

2

Robert Cupisz

Tech Lead, Demo Team


Lasse Jon Fuglsang Pedersen

Sr. Software Engineer, Demo Team

We’re presenting the work of Unity Demo Team. In the past we’ve made demos such as The Blacksmith, Adam and Book of the Dead.



3

In the past demos, we needed to implement basic building blocks of higher-fidelity rendering: area lights, volumetrics, soft shadows, some post-processing.


4

The Heretic

A work in progress

Now the HD Render Pipeline and Post Processing are quite solid, and we were able to put much more effort into building on top of that foundation. Among other things, we put some effort into human faces and vfx-heavy characters.


Let’s watch a preview we’ve released so far at GDC.



5

Screening, 3:45


Constructing the digital face for ‘The Heretic’

Lasse Jon Fuglsang Pedersen

Sr. Software Engineer, Demo Team


Some numbers

~28k vertices

~50k triangles

318 blendshapes

212 frames of 4D animation

~7 seconds

Base texture data

4K maps for albedo, normal, cavity, thickness, etc.

Pose-driven texture data

48 activation masks (some idle)

16x3 additional 4K maps for albedo, normal, cavity


Composition


Skin

Composition


Skin

Eyes

Composition


Skin

Eyes

Teeth

Composition


Skin

Eyes

Teeth

Eyebrows

Composition


Skin

Eyes

Teeth

Eyebrows

Eyelashes

Composition


Skin

Eyes

Teeth

Eyebrows

Eyelashes

Stubble

Composition


Skin

Eyes

Teeth

Eyebrows

Eyelashes

Stubble

Tearline

Composition


We’re relying heavily on HDRP

Lots of nice things out-of-the-box

With some customizations

Shaders for skin, eyes, teeth, tearline

Aiming to get these into HDRP

Shading


Based on HDRP/StackLit

SSS

Secondary specular

Adds pose-driven features

E.g. add wrinkles when the character squints

Specular occlusion in cavities

Also modulates smoothness

Fresnel fade

Skin shader


Emulates two-layer material

Geometry defines upper layer

Virtual plane for iris

Evaluates lightloop twice

Specular from upper layer

Diffuse from bottom layer

Adds “Light transform” concept

Used to light the iris

Marker-driven occlusion

Anisotropic spherical gaussian

Eye shader


Emulates two-layer material

Geometry defines upper layer

Virtual plane for iris

Evaluates lightloop twice

Specular from upper layer

Diffuse from bottom layer

Adds “Light transform” concept

Used to light the iris

Marker-driven occlusion

Anisotropic spherical gaussian

Eye shader

Nicholas Brancaccio


Teeth shader

Based on HDRP/Lit

SSS

Marker-driven occlusion

Area of spherical polygon

6 markers on the lips

Project onto hemisphere

Compute area (spherical)

Use area to fudge occlusion

Also applied to tongue/interior

Different diffusion profile


Tearline

Done with local blur

Normals, smoothness

Skin and eye shaders aware

Write to normal buffer in prepass

Read back before shading

Mark region with custom decal

Writes “blur normals” bit

Separate pass does actual blur

Works only on stencilled region

Kernel shrinks to edge


Animation

Wanted to have realistic facial animation

Attempt to avoid uncanny valley

High granularity of motion

Decided to try using 4D capture and playback

4D meaning one 3D scan per frame

“Geometry-accurate to actor’s performance”

At least macro-level

Brings a whole bunch of technical challenges...



4D technical challenges

Raw 4D data

“Photo-realistic” skin geometry

Missing sections and fine features

Cleaning the data

Retopologize to match neutral

Transplant missing parts

Partial noise removal

Recovering fine features

Correlate 4D frames with facial rig

Pose-driven normals, cavity, etc.


4D data pipeline

Neutral + FACS

4D capture

Clean 4D

Combine

Snappers

snapperstech.com

russian3dscanner.com


http://ir-ltd.net/

Infinite Realities

Russian3DScanner

Actor

Facial rig


4D technical challenges

Raw 4D data

“Photo-realistic” skin geometry

Missing sections and fine features

Cleaning the data

Retopologize to match neutral

Transplant missing parts

Partial noise removal

Recovering fine features

Correlate 4D frames with facial rig

Pose-driven normals, cavity, etc.

What about other parts, like hair?


What needs to move with the skin?

Hair

Eyebrows

Eyelashes

Stubble

Decals

Tearline blur region

Markers

Markers on eyelids

Markers on lips


What needs to move with the skin?

Hair

Eyebrows points

Eyelashes points

Stubble points

Decals

Tearline blur region points

Markers

Markers on eyelids points

Markers on lips points


Skin-attachments

We need to resolve lots of points

Everything that moves with the skin


Skin-attachments

We need to resolve lots of points

Build list of local poses for each point

Find closest vertex

Local pose per incident triangle

Barycentric + distance to plane


Skin-attachments

We need to resolve lots of points

Build list of local poses for each point

Find closest vertex

Local pose per incident triangle

Barycentric + distance to plane

Resolve at runtime

Batch operation

No dependencies between points

Parallel operation

C# jobs and Burst


Skin-attachments

We need to resolve lots of points

Build list of local poses for each point

Find closest vertex

Local pose per incident triangle

Barycentric + distance to plane

Resolve at runtime

Batch operation

No dependencies between points

Parallel operation

C# jobs and Burst

~429k poses, ~73k points


Skin-attachments

Longer strands require a bit more

E.g. eyelashes

Need them to pivot around roots

Locate roots of strands / cards

Find closest vertex at root

Propagate to tip

Runtime resolve not affected


33

Concept art by Georgi Simeonov

Robert Cupisz

Tech Lead, Demo Team


Boston is the name of this robotic companion. These are concept drawings by our concept artist, Georgi.


34

Concept art by Georgi Simeonov

From the beginning it was pretty clear we wanted a mechanical, robotic creature. Built with tech not available today, very physical in nature and reconfigurable.


35

Initial exploration - hacked together, slow, not very controllable...


36

But enough to suggest it might be a good direction.


Boston forms

37

Bird

Let’s see what were all the various forms we needed to support so far:

The bird form


Boston forms

38

Bird


Boston forms

39

Bird

Tentacle


Boston forms

40

Bird

Tentacle

Tentacle - crawling


Boston forms

41

Bird

Tentacle

Tentacle - crawling


Boston forms

42

Bird

Tentacle

Tentacle - crawling

Tentacle - leaping


Boston forms

43

Bird

Tentacle

Tentacle - crawling

Tentacle - leaping

Gauntlet


Boston forms

44

Bird

Tentacle

Tentacle - crawling

Tentacle - leaping

Gauntlet

Dreamcatcher


Tentacle form

45

Needed precise animator control, grabbing things. Couldn’t allow the wires to just flow anywhere.


Tentacle form

46

Hermite spline

If we say the wire should go through chain joints and be controllably smooth, we can solve its shape with a cubic Hermite spline.


Tentacle form

47

Hermite spline

Offset joint position

Then offset joint positions to get a bundle of wires. The plane of the circle is the mirror plane of the two bones. The rotation can follow the bone (as in the gif) or be automatic.


Tentacle form

48

The bundle behaves quite well under bends and twists.


Tentacle form

49

Start/end trim

Noise

Radius variation curve

...

Some extra controls to add variation: noise; distribution (Fibonacci, random, edge); trim at the ends.


50

Tentacle form

Initial exploration experiment with one of the wires made as a chain of HDRP line lights (and with an emissive material).


No simulation

51

No persistent state or reliance on previous frame

Better directability

Instant preview, exploration

Scrubbable

So far we’ve managed to avoid simulation completely. And by simulation I mean the current frame being a function of the previous frame. Another way of looking at it is reliance on some persistent state. Simulations add another layer of difficulty to making a complex effect directable, and making Boston directable in its form and animation was our biggest problem.

On top of that simulations are not scrubbable - require prewarming or caching. So the self-imposed limitation we had was that if we just now the positions of all the controls and values of all the parameters, it’s possible to evaluate Boston’s shape.


Wire rendering

52

Pre-generate a static tube mesh, store segment id and vertex angle in pos.xy

CS: Animate a segmented line*

CS: Generate segment tangent frame

Render instantiated wire meshes

VS: vertex knows its mesh instance, and segment id and vertex angle; indexes into segment position and tangent frame CS output buffers



* only this stage knows about wire behaviours

This is how wire animation and rendering is organised. There’s a static tube mesh that gets generated on init. Remaining stages run every frame. First all the animation compute kernels run on a buffer of points of a given wire, across all wires. Next stage, based on these output segment positions, generates a tangent, normal and binormal for each segment, outputs to a buffer as well. Wire meshes are rendered as instantiated geometry.


The regular HDRP Lit shader has a modded vertex modification stage. Since each vertex knows which segment it belongs to, it can look up segment position and tangent frame in the buffers. Each vertex also knows at what angle in the tube cross-section should it be. Based on that and the normal/binormal pair, the vertices are exploded into the final deformed tube shape. The animation compute kernels are the only ones knowing anything about all the different wire behaviours.


Tentacle crawling form

53

Need a way to conform to surfaces, collide with obstacles, but keep moving forward along the directed path.


Tentacle crawling form

54

This is a pretty good example of conforming to the surface and forming interesting shapes while at it.

But you can also see on the backside of the column how the wire paths quickly degenerate into a single wire. Still useful in some cases.


Tentacle crawling form

55

A better example of the wires remembering to continue

TheHeretic MakingOf DigitalDragons
Info
Tags Animation, VFX, Films, Animation technology
Type Google Slide
Published 23/09/2020, 16:15:08

Resources

Animation/VFX/Game Industry Job Postings