Text Files are Underestimated

Computers eat data in binary, that’s a fact, even if we have a text file the computer treats it as binary, zeros and ones. We can agree that binary serialization is the best in terms of disk space and load/save speed. But, what about the quality of life of using them? What about the time needed to add or create new functionality for such files? And the bugs and hard debugging that binary load/saving can give you…

I think that in some cases is best to just stick to text files, at least until other format is needed. It provides several benefits that from my point of view are underestimated.

Example

When I was designing the engine for my Junior year at DigiPen Bilbao, the engine that then would run Teotl – Rise of a God. I wanted to achieve fast iteration time in content creation, this is why I decided to implement hot reloading of resources, which allowed us to see the results of data changes instantly in the engine. If an artist changed a model or texture it would see the change in the running game at the moment.

On top of that, I wanted to achieve fast development time of engine/game features. For this, I decided to avoid implementing some tools and functionality to avoid having to invest the time it would take to develop them in first place and then improving and/or bug-fixing over the duration of the project. One of the features that we didn’t implement was an Entity Editor. As we where storing the data in Json format, thanks to the hot reloading of the Resource Manager we could iterate fast on the creation of entities.

This choice might seem as lazy, but indeed we didn’t miss this feature at all. Most of the times is not scalable but for us, having limited time to develop a full 3D engine and game, it was a good choice. A text editor did the job, we didn’t have to spend time working on it and it was 100% reliable, it didn’t crash even once 🙂

Outsourcing functionality to text editors

Developing cool tools with cutting edge features is very tempting, I’m always looking for some challenges in my day to day programming. But the truth is that the more cutting edge we try to do something the more time it will take, more code will be needed, more potential for bugs. If we need to create some UI for this feature, we also need to make sure that is intuitive and easy to use for end users.

On the other hand, using text files to represent our data allow us to use them as our editors, inheriting all their features:

* Search: Instead of creating a database, a text file and Search functionality in text editor does the job (most text editors support regex and other fancy ways of searching, no need to go through the headache of implementing them yourself! : )

* Find and Replace: Easy to update data.

* Stats: By using the search functionality you can find out how many times a text appears in the file. That “text” could be the name of a property, the path to a resource… Visual Studio Code highlights on the search bar where the occurrences are, this can also give you data (i.e. in a file where you have a dump of all your levels, you can find out how often a specific mesh is used across levels).

We inherit all benefits of text editors! By default!

Mike Acton likes things “by default” ¹

All of the above happens by default, without any extra work required from us or any other person in our team. To make it even better, we also get these benefits:

• Minor or no need to learn how to use the editor (people is already knows how to use text editors).
• Bug free (hopefully).
• Free lifetime updates with new features they add to the editor that can be handy.

These might sound like they are not much, but taking into account we get them without spending any of our time, they are worth considering.

Building ‘true’ standalone tools

Storing the data in binary format makes it tied to the program/API that is producing the data, this means that custom tools we build to work with our data will only work with that, fullstop. It can be “generic” in the context of the project we are working on, but it will be “specific” to that project in the context of other projects.

As programmers we are always looking for ways to make our code as standalone as possible, looking for re-usability. By having our data as text we can build custom tools to search, update, find statistics, etc; that latter can be reused in any other place, because we are just building tools to work with text.

Batch files

Sometimes we need to perform operations on several files or even all our files and doing it by hand (in a text editor) is not an option. In those cases we can create some simple batch file that performs the needed operation in all the files we want. By doing this we again have more tools at our disposal like findstr or grep. Once again, by default!

This batch files could be latter reused on other projects, it doesn’t matter if they use plain text, xml, json… they just work with text.

Programs

In case we need a more custom operation, we can always create a simple program that would perform it. This program can should be completely standalone from our project, it could be written to work with plain strings or using the API for the format we use (i.e. json/xml). This tools work just with some data and the operations they perform can be reused across any project, it can be a game, an emulator, some other tools or even it could be used to work with code! Isn’t it great?

Real World Example

Naugty Dogs uses a lot of text files to develop multiple of their features (i.e. dialog system, animation metadata…).

This saves work on the programmers side because there are no tools to be developed. It might seem like the quality of life of users is decreased because they don’t have a UI to work with, but the truth is that having no tool also means avoiding crashes and potentially loosing work.

If you are interested in this topic, Jason Gregory, exlains how their dialoges are created using Scheme in his book Game Engine Architecture ². Additionaly, you can see a brief mention of Scheme sintax and how they use it in this talk: HandmadeCon 2016 – Large-scale Systems Architecture ³.

Final thoughts

Sometimes, rather than spending a lot of time thinking on how to build some new tool, is worth asking the question “Would a simple text file do the job?”, if the answer is “Yes” we have the opportunity to save time and open a huge range of possibilities.

I think that most of the times we overlook text files as a viable option, maybe is because using text files feels “naive”, maybe is because the dream idea we have in mind blinds us. We can always find thousands of unrealistic scenarios where “a text file won’t work”, we are always looking for answers to the “What if…” question that will reasure us that we need a super custom fancy tool. “What if…

• … we have too much data? It will be slow.”
• … we want to compare values? A text file can only search exact matches. Using regex is difficult.”
• … we mess up the format of the data with a find and replace?”
• …

My experience tells me that most of the “What if…” questions we make ourselves don’t turn out to happen, instead, other new problems arise and they need to be solved afterwards. I’ve found the features provided by text editors very useful along my career, if the problem needs a more complex solution, starting with text files as intermediate step shows if an other solution is really needed ⁴.

References

1. When I wrote “We inherit all benefits of text editors! By default!” it was a reference to Mike Acton talk about Performance by Default in Unity.
2. Book: Game Engine Architecture by Jason Gregory (chapter about their dialogue system).
3. HandmadeCon 2016 – Large-scale Systems Architecture: Jason Gregory explains how they use a Scheme based text format for different purposes.
4. Using text files as a first step towards developing a more complex feature is also aligned with on of the rules Mike Acton points out in his Everyone Watching This Is Fired talk: “I have implemented my plan B in case my solution to my current problem doesn’t work” -> “I can work with my data in text files in case the Editor doesn’t work or there is no time to finish it”.