Thanks. My article already refutes all this. But it’s in my queue to blog about here eventually.

Speed is one thing it can do. But stylo just runs software on a database. We already have the software and the database and people have already run it. So AI would not improve on that. Hence AI would just be a quality-sucking middleman to the process. Experts can just do the math directly and not have to trust AI for anything. It’s always better to actually do the thing than farm it out to an unreliable intern.

Nevertheless, maybe someone can come up with some clever way to improve on that prospect. But it would be a feat.

Yes, I know, AI is hallucinating too often. Which means, AI acts like humans do all the time, and they won’t admit it, like AI. I don’t think it can replace stylometrics, but it can explore a lot of text faster than humans — humans still need to investigate the results, though.

That has more or less been done. It’s (in a more statistical-sampling way) what the studies validating stylometric procedures that I cited do.

Except the AI move. That’s too unreliable. AI hallucinates too often. And it buries the actual data you need. It’s like asking a high school student to summarize a dataset: you’d be better off just looking at the dataset yourself.

Examples come from the AENEAS project, which uses LLM to reconstruct text on damaged papyri: it does accelerate the process, but does not work by itself, as experts have to constantly correct it. And there, the experts can go look at the raw data themselves to vet the results. So all the LLM does is speed a process; it does not do it better. I doubt it would be of any help in stylometrics. That’s a method that works better by removing the middleman, not adding one.

1. Get plenty of texts that are written in old Greek.

2. Remove all quotations from the texts. 

3. Split every document in half, then again in thirds, then in fourths, until each portion is too small to run an analysis. Give each split document a random, non-repeatable name. You can split documents in different ways: take the first half and the second half. Take the first paragraph for text #1, the second paragraph for text #2, the third paragraph for text #1, and so on. And you can even randomize the distribution of sentences or paragraphs. Which means, from a few texts, you can create thousands of variants!

4. Create a database of all documents that contain the name of the author and the date (if possible), and link it to the random name. We now have texts that we know are from the same author.

5. Run a stylometric analysis on all documents. You will get some false positives (texts that are not from the same author) and some false negatives (texts that are from the same author). Now you can estimate how accurate the analysis was by comparing each document to every other document. 

6. Create another program that will randomly change the parameter set used in the stylometric analysis. Repeat step #5 for every new set. Of course, you can change the method for all stylometric analyses, from simple ones to more complex methods.

7. Count the false positives and the false negatives for each set (and method) until you find the most reliable parameter set (and method). Notice that you can even determine if texts are too small for the analysis (the false positives and negatives will go up with smaller texts).

8. You can create hand-optimized sets as well. You might notice a pattern: when you make the value of a parameter larger, the number of false positives and negatives goes up; making it smaller will reduce the false results. Run all analyses until the number of false results does not improve (get smaller).

9. After you find the best parameter set, you know how many false negative and false positive results you get. You know, too, when the documents are too small to get a valid result.

10. Now you can run the stylometric analysis with the best set found so far. Notice that you can distribute the analysis on different computers to speed things up. And you can judge the overall reliability of the analysis.

The first two steps need a person who knows old Greek well, so I’m out; I can’t do that. For the rest of the steps, someone with knowledge in computer programming is needed; this person does not need to know anything about Greek! This is an evolutionary approach to finding the best method.

Of course, as always, it might turn out that a stylometric analysis of this kind does not work well. That would be too bad, but that’s science.  

Two additional steps might be useful:

11. Create a Large Language Model (LLM) for Old Greek.

12. Train it on the data you found that can decide whether a text was written by the same author or not written by the same author. You can use this AI to further investigate which texts are written by the same author (or come from different authors).

Plus, you can try to insert into one text one or a few paragraphs from a text from a different author. What does the stylometric analysis tell you now? Can the AI detect that the text was tampered with and contains paragraphs from another author? How does this affect the reliability?

I think you can get a Ph.D. for doing this, but it is a lot of work.

But that might create a useful tool for analyzing old documents (it must be repeated for every different language, though). If stylometric analysis is viable, you will find out—or that this is all garbage and can’t be used. There will be no more guessing how reliable stylometric analysis is.

If you want to go a long way, you can even use two or three on all the texts. It might turn out that this will enhance reliability if you combine different methods. But you will need plenty of computers for this, something like “SETI at Home.”

I’m unpersuaded. He notes that second century authors mention in passing Claudius permitted a Cybele-Attis cult yet his evidence for the myth span from the late second to the fourth century, synthesising them into one picture. I found it surprising such a slingshot approach was published in a University press or am I being too naive to expect a Historian not to be this speculative?

Big hat, vague accusations, no details, not even names. Claims “many” have repeated their results. We may patiently await publication of those others’ results.

No apology warranted. That was a good question, not least because it’s on an important point usually glossed over so I was happy to have it, but also simply because I’m a historian, and answering neat questions about my field are my favorite thing.

But yes. The Roman Empire has a lot of horrors. Just like the British Empire. They both had a lot of positives and progress in them too (e.g. within the history of civilizations up to then, slaves and women were never better off than under the Pax Romana, and yet were still in a shit position ten ways from Sunday). But it’s still all very backwards compared to now—and still unmistakably fascism.

My Columbia University dissertation advisor put it like this: “Imagine if the old-school Italian Mafia took control of the entire U.S. government, all three branches. You will be describing the Roman Empire.”