The final appearence using lighting and all the different methods discussed below
This project has been going on for a while now. I've worked on it from time to time when I learnt more about unreal and it's features.
My first attempt at outlines was to used an inverted hull, it gives the desired effect of giving objects outlines. But it doubles the mesh of all objects. and since I was the 2D, 3D and VFX artist on this project I didn't want to have to model specifically for getting good inverted outlines.
I tried out the same methods I had been using for outlines when working in blender. Trying to extract the angles on objects and adding lines based on that. But I quickly noticed it doesn't work in a living and moving environment. So I scrapped it quickly.
(Tried making hatching with textures, also scrapped)
I tried making outlines by simply adding a small zone between the different shades in the cel shader to make within shapes. My idea was to combine this with my edge detection to get a good line art for my shader. This did not work out so well...
Here you can see why it didn't work. It only works for specific shapes. When the shadow becomes low contrast. There outlines can become huge and no longer feel like outlines.
I looked at a lot of Moebius artworks to help me in the search of simplified ways of giving the impression of lighting and line weight. I ended up using a gradient based on world position to give the impression of a top down lighting without the actual use of any lighting. and then I use the diffuse colour of the objects to define the lineart in the shader. I base the lineart on the actual screen. I did it this way because line art is often done to adhere to the viewers ability to read the lines. If the viewer will not be able to percieve a line then there does not need to be a line there. So by using the view of the player. I can add the line work where it actually matters. And in the process of doing so, we also get distance scaled lines for free. This process also gives the artist a lot of room for artist expression since the shader takes into consideration both texture and modeling work.
The final method uses convolution written in HLSL and takes the inputs shows in the image. The scenetextures can be any scenetexture since they are just connected to access scenetexture components in the HLSL code.
I added shadow edge detection and a paper texture onto the camera to make this final appearance.