Conclusions
Before making any claims about the “accuracy” of the algorithm’s determinations here I would naturally want to consult with an expert in the literary field, of which I am not. Still I think it’s safe to say that the results here appear to be mostly random and arbitrary nonsense, with occasional bouts of comic fun (e.g. the “Benjamin Franklin” result). While I did not yet run extensive tests here to see how well the AI properly identified an author’s own work, the tool overall appears to be more of an academic toy than a serious piece of technology with plausible real-world applications.
Still, I think there is a good argument to be made that if I had taken this AI app more “seriously”, and entered in paragraph-long results (or even sentences that attempted to be more in line with a “literary style”) the results may have been more “accurate.” I think that is a good argument to make, but an obvious counter-question would be: “What is writing style but punctuation and words selected and arranged in a somewhat arbitrary fashion?” The app, after all, claims to be a “style predictor”, and states its predictions with complete confidence.
Related, during my testing I openly wondered if the results for some entries (such as the one for “a b c d e f g h i”) were hard-coded into the algorithm as some kind of a joke. However, on further analysis there is actually good reason to believe that results for this are coming from the algorithm directly, and were not due to some kind of manual override. I say this because while entering “a b c d e f g h i” (without quotes) gives you Leo Tolstoy, entering “a b c d e f g h i j k” (also without quotes) returns Marcus Aurelius, and entering “a b c d e f g h i j k l m n” gives you Lewis Carroll. As further evidence that this isn’t a hard-coded joke, a search for the text string “a b c d e f g h i” on Project Gutenberg shows that these letters—for unknown reasons—give you a link (among other texts) to Leo Tolstoy’s novel “Anna Karenina” (see: https://www.gutenberg.org/ebooks/search/?query=a+b+c+d+e+f+g+h+i). I have no real insight at the moment as to why.
It’s worth noting that in past eras a web application like this might have been posted on April Fools as a joke or a hoax and then removed shortly thereafter. Here however the Peltarion tool—trained with sample data dated November 11, 2019—is presented as a serious AI product, and contains no disclaimers about its accuracy or lack of accuracy; the tool’s methods appear to be fully representative of the capabilities of Peltarion’s BERT neural network architecture, and of the Neural Network approach in general.
Clearly, there has to be a better way of handling this entire exercise; one that actually provides both meaningful stylistic feedback and way to cross-analyze different author’s styles in a quantitative manner. And luck would have it, I’ll be exploring some ways to do exactly that in my next post.
Stay tuned…
Sincerely,William Gerhardt James Joyce
P. S. “James Joyce” is the actual result I got by feeding the entire last paragraph of this article (including the text “stay tuned…”) into the application. However, trivial word changes made during editing caused the paragraph to fluctuate between a few different authors including “Herman Melville” (one prior version), “Lucy Maud Montgomery” (two prior versions), and “James Joyce” (in five other edited versions). Again, the author determinations here appears to be very fickle, because the paragraph remained fundamentally unchanged in “style” throughout. It appears most likely the algorithm is making its style determination based on the use of specific words and punctuation markers, and nothing else.
