Game assets, with AI...
edgeSay, I want to convert a nice 1920's Bowman live-steam loco into my game. How could we do this? Well, first check if someone else did it and is willing to share the model. If not, I have to find my own workflow.
According to the web there are a lot of different routes, generating amazing models, so let's find a process with the least amount of effort and the perfect results...
In short
- Most accurate: a full CAD model of the shell, with Claude (in Blender) generating the textures and decals. Perfect results, but about 8 + 8 hours per model.
- Quick, for less important objects: TripoAI SmartMesh with 4 photos. About 5 minutes and ~18€ a month, good enough for assets further away.
- Photogrammetry (Meshroom, RealityScan): a shiny toy locomotive is hard. A turntable only works when the background is masked out; after that the result is average and useful as an underlay. See 2a.
- 3D scanners: expensive and still time consuming. Not convincing yet.
The goal
Generating multiple assets, considering 3 goals:
- Going close to reverse-engineering. I want to have an accurate model in CAD. But I'll not add screws, rivets etc. 'Only' the outside shell should be accurate, including textures.
- The asset should be able to be shown, like on the SketchFab platform. Preferably with animations and exploded views, so it might be used as a demonstration model.
- The asset should be usable within movies & games. Basically downgrading the SketchFab model further, changing 4K textures into 1K (mobile platform) or 2K (PC-like games).
The subject is an original Bowman live-steam locomotive I own. Unfortunately it's not working, it needs some soldering etc. Well, that's something for during retirement. The nice thing about this: I can measure everything, which will help a lot. For now it should take less effort, getting a 'virtual working' locomotive.
Basically 2 sources: one nice looking loco and my own (which is very handy for measuring).
Preparation
For good photos, good light conditions are necessary. So while trying different methods, the method for taking photos is also improving along the way. Here the different light box versions are shown.
LBv1: Light box version 1
First I needed some bright pictures, so let's take them outside. It's cloudy but still relatively bright.
LBv2: Light box version 2
For all photogrammetry & 3D scanning options we need proper light conditions. Let's start building a light box.
The current issue is creating light diffusers: the backing paper is not working great, so currently all scans have been done without.
Let's start generating 3D assets...
Although the main subject is the Bowman train, other models are used during this process too.
1. Simplified CAD model (no textures)
Just do it yourself: measure every part & detail and model it in CAD. Well, I was hoping this would not be necessary.
So let's start with a simplified CAD object and use this as input for some of the methods below.
2. Photogrammetry: photos → 3D model
Let's try photogrammetry: take a bunch of photos from different angles and let the PC calculate the 3D model. This is proven technology, but it does not work so well with shiny objects. That's why I also built a hobby-grade light box, to reduce reflections.
2a. Meshroom
My first photos were not bright and Meshroom had problems recognizing the model. After 5 runs the picture is clearer, and the light was not even the main problem.
What went wrong: I put the locomotive on a rotating disc and kept the camera still. The light box walls also stand still, so Meshroom sees a static background that says 'the camera did not move' while the train says 'the camera moved'. Meshroom assumes a static scene, so it only managed to place 39 of 46 photos, scattered around.
What helped, step by step (number of photos where Meshroom could calculate the camera position):
| Run | Setup | Cameras placed |
|---|---|---|
| 1 | Standard pipeline, 46 photos | 39 of 46 (scattered, torn model) |
| 2 | Turntable pipeline: background masked out | 21 of 46 |
| 3 | + feature extraction preset 'High' | 37 of 46 |
| 4 | 2 photos with a bad mask removed | 37 of 44 |
| 5 | 83 frames from a video of the turntable | 67 of 83 |
- Mask the background. Meshroom has a turntable pipeline (photogrammetryObjectTurntable) that cuts out the train and the marker disc first. Check the masks, a bad mask is worse than no photo.
- Masking on the CPU. The AI masking models did not run on my RTX 5060 (the bundled PyTorch has no kernels for RTX 50 cards: 'no kernel image is available'). Setting useGpu = False for the two masking nodes solves it; the rest of the pipeline still uses the GPU.
- Use a video. Frames from a video of the turning disc give small steps between the images, so Meshroom can link them. A path without spaces helps too (the batch launcher splits arguments at a space).
- Clean up in Blender (script). Find the plane of the disc, level the model, cut the disc off, remove floating fragments and reduce 840,000 faces to 200,000. The GLB went from 94 MB to 9 MB. The scale is arbitrary; set it from a measurement.
Result: the whole locomotive is there, the boiler bands, wheels, rods and the LMS lettering are recognizable. The cab roof is thin and ragged, the cab rear and the underside have holes, and the texture is soft. Average: good as an underlay or for a distant asset, not as the final model.
2b. RealityScan
The same photos are also used for RealityScan, which also has problems recognizing the model.
I took 5 iterations and will quit for now. Since Meshroom had the same problem, the same cause (a still background around a turning object) probably applies here too. First I need a better light box.
3. 3D scanning
3a. Xbox Kinect
I used this scanner in the past (around 2016) for scanning people's heads and 3D printing them. That was working.
I already knew it's quite difficult to use these kind of scanners on shiny surfaces, because of the reflections. Installing this thing was possible on Windows 11 too, but took a lot of effort and scanning the locomotive was impossible. This is definitely a no-go.
3b. Revopoint POP 4 (LBv2)
Let's try the new Revopoint POP 4 scanner. According to the supplier, the blue laser should be able to scan metal parts too. It scans quite fast and is able to measure colors too. Let's try...
(I tried another model for now, see below for the comparison. The Bowman train I'll do later, when I have a bit more scanning experience.)
It takes a lot of time - although partly automated - and result is (not yet) great.
4. TripoAI: image → 3D model + textures
Using TripoAI would be great, let's see how smart AI is today.
4a. TripoAI (free) 1x photo only
Using the free TripoAI with one photo. It's working, but of course a lot is missing and the text decals are mirrored and not readable. Of course this is not working well.
(No picture: the model was not usable.)
4b. TripoAI (free) Simplified CAD + 1 photo
It's also possible to upload your own 'mesh' and add 1 photo for free.
One of the problems is the text on the smoke box. This is wrongly 'written' on the other side. For simplified assets, this might be an okay direction.
4c. TripoAI - HD Mesh + 4 photos
Now using a paid TripoAI account, enabling 4 side views.
I basically used the wrong approach. An 'HD Model' might generate nice results, but it's not working for this train model.
4d. TripoAI - SmartMesh + 4 photos
Same as 4c, but now using the 'Smart Mesh' option. This seems to be a better model, converting meshes right!
It's an acceptable way of generating less important or further away objects/assets.
4e. TripoAI - Simplified CAD + 4 photos
Okay, let's use the simplified mesh again and let TripoAI generate the textures.
As expected, the more detail/correct geometry you add, the better the model.
5. Recreate: CAD → Claude + Blender
Let's move back to the simplified CAD model and let Claude control Blender:
5a. Simplified CAD + Claude merge photos
Claude is now in control of 2 things:
- Creating some minor missing 3D solids
- Projecting decals from the photos (photogrammetry output)
The results are still very bad. Although small objects could be generated, it still takes time. And re-using these photos is not working at all.
5b. Simplified CAD + Claude creates solids + textures
Claude recreates the textures in Blender. These are lines of code, telling Blender what to do.
I should have created more shapes inside CAD, since (not accurate) modeling takes more time writing prompts than just drawing it. Overall Claude is quite helpful in changing many things, like changing colors and adding real small missing details.
Almost perfect model:
5c. TripoAI 3D template as underlay
Let's improve workflow 5b:
- First use TripoAI, generating a first 3D model.
- Import the blue STL model in CAD and scale it (using this as underlay!).
- Now start modeling the full CAD model (except for the driver, that's no fun doing in CAD).
- Let Claude generate textures & decals via Blender.
(I also need a new subject for a third Revopoint trial...)
TripoAI was not generating straight surfaces. It was not bad, but just not good enough, so the CAD modeling still needed to be done. Since I'm borrowing this car model, measuring is better than guessing, so finally I did not use the underlay model at all. It's still a feasible way if you only have pictures.
5d. Full CAD model + Claude & Blender for texture & decals
Instead of letting Claude finish the 3D model, it's easier to do this in CAD. So let's try this with a new model.
- First I needed some bright pictures (LBv1).
- Then I let Claude learn a skill to add textures and decals: textures are basically standard textures & line segments, where decals are bitmaps.
- Modeling took 8 hours, textures & decals also took 8 hours.
Perfect 3D model:
Let's summarize
| Method | Effort | Time & cost | Result |
|---|---|---|---|
| 1a. Simplified CAD model | Medium | 1h | POOR, textures missing |
| 1b. Full CAD model | HIGH | 8h | GOOD, textures missing |
| 1c. Full CAD model, including textures | HIGH | +8h | PERFECT (not tried) |
| 2a. Meshroom * | Medium (many photos) | ~2h | AVERAGE (shape OK, holes, no real scale) |
| 2b. RealityScan * | Medium (many photos) | ~2h | POOR |
| 3a. Xbox Kinect | Medium (long scanning) | +2h & 15€ | N/A |
| 3b. Revopoint POP 4 * | Medium (long scanning) | +2h & 750€ | AVERAGE (different model) |
| 4a. TripoAI (free) 1x photo only | LOW | 5 min | POOR |
| 4b. TripoAI (free) Simplified CAD + 1 photo | Medium | 1h + 5 min | POOR |
| 4c. TripoAI - HD Mesh + 4 photos | LOW | 5 min & 18€/month | POOR |
| 4d. TripoAI - SmartMesh + 4 photos | LOW | 5 min & 18€/month | AVERAGE (model not exact) |
| 4e. TripoAI - Simplified CAD + 4 photos | Medium | 1h + 5 min & 18€/month | AVERAGE (simplified model) |
| 4f. TripoAI - Full CAD + 4 photos | HIGH | 8h + 5 min & 18€/month | AVERAGE (textures still odd) |
| 5a. Simplified CAD + Claude: merge photos | Medium | 1 + 3h & 20€/month | POOR |
| 5b. Simplified CAD + Claude: solids + textures | HIGH | 1 + 8h & 20€/month | PERFECT (Bowman not finished) |
| 5c. TripoAI CAD underlay + Claude: textures | Medium | ~4 + 8h & 38€/month | AVERAGE (model not exact) |
| 5d. Full CAD + Claude: textures | HIGH | 8 + 8h & 20€/month | PERFECT (different model) |
*) Effort 'medium': especially for these methods, shiny objects are difficult to cover. For the Revopoint POP 4 the supplier claims the blue laser would help with this... So a good light box is important. In general: photos can just be taken with your phone, so no extra costs.
In short, the conclusions (for these type of shiny objects):
Perfect results:
- 5d. Full CAD model, let Claude produce textures.
- 1c. Full CAD model, including adding textures yourself (which software to use?).
- (5b. Simplified CAD model, let Claude create solids & produce textures.)
Simplified game assets:
- 4d & 4e. Creating quick but less accurate models.
- 2a. Meshroom, if you take care of the background (masks) and use a video: a usable underlay.
- 3b. Maybe good enough, but costly and still time consuming.
Compare figures: Revopoint POP 4 versus TripoAI
Let's convert a figure into 3D:
- 3b.: 3D scan with the Revopoint POP 4.
- 4d.: Feeding TripoAI, using 4 photos.
(Pictures to be added.)
Of course TripoAI cannot see the complete model with 4 pictures, so it will fill holes with fantasy. But this is real quick and quite accurate, generating a decent amount of faces. The cost is only ~18€ a month. 3D scanning has the same problem: you need a lot of scanning at different heights to show the complete model, which is also the case for the Revopoint POP 4. I'm not yet seeing the 750€ value... Hopefully a good scan takes less time than methods 5b/5c. (To be continued...)
5d. Adding more
Just for the record (hours spent per model):
| Model | CAD [hours] | Claude + Blender [hours] | Claude weekly limit [%] |
|---|---|---|---|
| Bowman 300 (5b) | 1 | 8 (not finished) | ? |
| Hessmobil 8036 | 8 | 8 | ~20 |
| Hessmobil 5003 | 6 | 6 | 15 |
| Hessmobil 1637 | 5 | 6.. (busy) | 15.. |
| Hessmobil 1639 | 5 | ToDo... | |
| Hessmobil 1022 | 6 | ToDo... |
