How do Turnitin’s AI writing detection capabilities work for Spanish?

 

Does Turnitin offer a solution to detect AI writing for submissions in Spanish?

Yes. Turnitin’s has released its  AI writing detection capabilities in Spanish to help educators uphold academic integrity while ensuring that students are treated fairly.

We have added an AI writing indicator to the Similarity Report. It shows an overall percentage of the document that AI may have generated. The indicator further links to a report that highlights the text segments that our model predicts were likely written by AI. Please note, only instructors and administrators are able to see the indicator.

Our AI writing detection model may not always be accurate (it may misidentify human-written text), so it should not be used as the sole basis for adverse actions against a student. It takes further scrutiny and human judgment in conjunction with an organization's application of its specific academic policies to determine whether any academic misconduct has occurred.

 

How does it work?

When a paper is submitted to Turnitin, the submission is first broken into segments of text that are roughly a few hundred words (about five to ten sentences). Those segments are then overlapped with each other to capture each sentence in context.

 

The segments are run against our AI detection model, and we give each sentence a score between 0 and 1 to determine whether it is written by a human or by AI. If our model determines that a sentence was not generated by AI, it will receive a score of 0. If it determines that the entirety of the sentence was likely generated by AI, it will receive a score of 1.

Using the average scores of all the segments within the document, the model then generates an overall prediction of how much text in the submission we believe has been likely generated by AI.

Currently, Turnitin’s Spanish AI writing detection model can detect content from the GPT-3.5 and GPT-4 language models. We are actively working on expanding our model to enable us to better detect content from other AI language models.

How was Spanish model trained?

Our model is trained on a representative sample of data spread over a period of time, which includes both AI-generated and authentic academic writing across geographies and subject areas. While creating our sample dataset, we also took into account second-language learners to minimize bias when training our model.

Does the Spanish detector identify AI paraphrased content when tools such as Quillbot, etc., are used to paraphrase AI-generated content?

Our Spanish detector is trained to detect content likely generated from GPT-3.5 and GPT-4, and modifying text generated by these systems will have an impact on our detector’s abilities to identify likely AI written text. While we recently released AI paraphrasing detection capabilities in our English AI detector, we’re currently developing similar capabilities for the Spanish model and they will be released in the future.

Can I check past submitted assignments in Spanish for AI writing?

Yes. Previously submitted assignments can be checked for AI writing detection if they’re re-submitted to Turnitin and if you have AI writing enabled for your account. 

What will happen if a non-English or non-Spanish paper is submitted? 

If a non-English or non-Spanish paper is submitted, the detector will not process the submission. The indicator will show an empty/error state with ‘in-app’ guidance that will tell users that this capability only works for English or Spanish submissions at this time. No report will be generated if the submitted content is not in English or Spanish.

Can my institution get access to AI detection to be able to trial it? 

No, new customers should speak to a Turnitin representative to get a demo of the capability. 

Can I or my admin suppress the new indicator and report if we do not want to see it? 

Yes, admins have the option to enable/disable the AI writing feature from their admin settings page.  Disabling the feature will remove the AI writing indicator & report from the Similarity report and it won’t be visible to instructors and admins until they enable it again. This will mean that submissions made to Turnitin will not be processed for AI writing detection, for both, English & Spanish papers.

Will the addition of Turnitin’s AI detection functionality to the Similarity report change my workflow or the way I use the Similarity report?

No. This additional functionality does not change the way you use the Similarity Report or your existing workflows. Our AI detection capabilities have been added to the Similarity report to provide a seamless experience for our customers. 

Will the AI detection capabilities be available via LMSs such as Moodle, Blackboard, Canvas, etc?

Yes, users will be able to see the indicator and the report via the LMS they’re using. We have made AI writing detection available via the Similarity report. There is no AI writing indicator or score embedded directly in the LMS user interface and users will need to go into the report to see the AI score.

How is authorship detection within Originality different from AI writing detection?

Turnitin’s AI writing detection technology is different from the technology used within Authorship (Originality). Our AI writing detection model calculates the overall percentage of text in the submitted document that was likely generated by an AI writing tool. Authorship, on the other hand, uses metadata as well as forensic language analysis to detect if the submitted assignment was written by someone other than the student; for example, a paper mill. It will not be able to indicate if it was AI written; only that the content is not the student’s own work.

AI detection results & interpretation

What does the percentage in the AI writing detection indicator mean? 

The percentage indicates the amount of qualifying text within the submission that Turnitin’s AI writing detection model determines was likely generated by AI. This qualifying text includes only prose sentences, meaning that we only analyze blocks of text that are written in standard grammatical sentences and do not include other types of writing such as lists, bullet points, or other non-sentence structures.

This percentage is not necessarily the percentage of the entire submission. If text within the submission is not considered long-form prose text, it will not be included. 

 

What is the accuracy of Turnitin’s Spanish AI writing indicator?

We strive to maximize the effectiveness of our detector while keeping our false positive rate - incorrectly identifying fully human-written text as AI-generated - under 1% for documents with over 20% of AI writing. In other words, we might flag a human-written document as AI-written for one out of every 100 fully-human written documents. 

 

The percentage shown sometimes doesn’t match the amount of text highlighted. Why is that? 

Unlike our Similarity Report, the AI writing percentage does not necessarily correlate to the amount of text in the submission. Turnitin’s AI writing detection model only looks for prose sentences contained in long-form writing. Prose text contained in long-form writing means individual sentences contained in paragraphs that make up a longer piece of written work, such as an essay, a dissertation, or an article, etc. The model does not reliably detect likely AI-generated text in the form of non-prose, such as poetry, scripts, or code, nor does it detect short-form/unconventional writing such as bullet points, or annotated bibliographies.

This means that a document containing several different writing types would result in a disparity between the percentage and the highlights.

What do the different indicators mean?

Upon opening the Similarity Report, after a short period of processing, the AI writing detection indicator will show one of the following:

• Blue with a percentage between 0 and 100: The submission has processed successfully. The displayed percentage indicates the amount of qualifying text within the submission that Turnitin’s AI writing detection model determines was likely generated by AI.

As noted previously, this percentage is not necessarily the percentage of the entire submission. If text within the submission was not considered long-form prose text, it will not be included. To explore the results of the AI writing detection capabilities, select the indicator to open the AI writing report.

Our testing has found that there is a higher incidence of false positives when the percentage is less than 20. In order to reduce the likelihood of misinterpretation, the AI indicator will display an asterisk and will show no score for percentages less than 20 to call attention to the fact that the score is less reliable.

To explore the results of the AI writing detection capabilities, select the indicator to open the AI writing report. The AI writing report opens in a new tab of the window used to launch the Similarity Report. If you have a pop-up blocker installed, ensure it allows Turnitin pop-ups.

• Gray with no percentage displayed (- -): The AI writing detection indicator is unable to process this submission. This can be due to one, or several, of the following reasons:
  • The submission was made before the release of Turnitin’s AI writing detection capabilities. The only way to see the AI writing detection indicator/report on historical submissions is to resubmit them.
  • The submission does not meet the file requirements needed to successfully process it for AI writing detection. In order for a submission to generate an AI writing report and percentage, the submission needs to meet the following requirements:

• • • File size must be less than 100 MB
    • File must have at least 300 words of prose text in a long-form writing format
    • Files must not exceed 30,000 words 
    • File must be written in Spanish
    • Accepted file types: .docx, .pdf, .txt, .rtf
• Error ( ! ): This error means that Turnitin has failed to process the submission.  Turnitin is constantly working to improve its service, but unfortunately, events like this can occur. Please try again later. If the file meets all the file requirements stated above, and this error state still shows, please get in touch through our support center so we can investigate for you.

 

What can I do if I feel that the AI indicator is incorrect? How does Turnitin’s indicator address false positives? 

If you find AI written documents that we've missed, or notice authentic student work that we've predicted as likely AI-generated, please let us know! Your feedback is crucial in enabling us to improve our technology further. You can provide feedback via the ‘feedback’ button found in the AI writing report.

Sometimes false positives (incorrectly flagging human-written text as likely AI-generated), can include lists without a lot of structural variation, text that literally repeats itself, or text that has been paraphrased without developing new ideas. If our indicator shows a higher amount of likely AI writing in such text, we advise you to take that into consideration when looking at the percentage indicated.

In a longer document with a mix of authentic writing and likely AI generated text, it can be difficult to determine exactly where the likely AI writing begins and original writing ends, but our model should give you a guide to start conversations with the submitting student.

In shorter documents where there are only a few hundred words, the prediction will be mostly "all or nothing" because we're predicting on a single segment without the opportunity to overlap. This means that some text that is a mix of likely AI-generated and original content could be flagged as entirely likely AI-generated. 

Please consider these points as you are reviewing the data and following up with students or others. 

Will students be able to see the results?

The AI writing detection indicator and report are not visible to students. However, with the PDF download feature, instructors can download and share the AI report with students.

Does the AI Indicator automatically feed a student’s paper into a repository?

No, it does not. There is no separate repository for AI writing detection. Our AI writing detection capabilities are part of our existing similarity report workflow. For institutions that have the AI writing feature enabled, when we receive submissions, they are compared and evaluated via our proprietary algorithms for both similarity text matching and the likelihood of being likely AI writing (generated by LLMs). 

If enabled, AI writing detection is run on a submission and the results are shared on the similarity report. Results regarding the percentage AI writing identified by the detector, along with the segments identified as likely written by AI – are retained as part of the similarity report.

What is the difference between the Similarity score and the AI writing detection percentage? Are the two completely separate or do they influence each other?

The Similarity score and the AI writing detection percentage are completely independent and do not influence each other. The Similarity score indicates the percentage of matching-text found in the submitted document when compared to Turnitin’s comprehensive collection of content for similarity checking. 

The AI writing detection percentage, on the other hand, shows the overall percentage of text in a submission that Turnitin’s AI writing detection model predicts was likely generated by AI writing tools. 

Does the Turnitin model take into account that AI writing detection technology might be biased against second-language writers?

Yes, it does. One of the guiding principles of our company and of our AI team has been to minimize the risk of harm to students. Hence, while creating our sample dataset, we took into account statistically under-represented groups like second-language learners to minimize bias. Internal tests have shown that the false positive rate for Spanish-language learners is less than 1% for documents with more than 20% AI content.

How can I use the AI indicator percentage in the classroom with students?

Turnitin’s AI detection indicator shows the percentage of text that has likely been generated by an AI writing tool while the report highlights the segments that may be AI-written. The final decision on whether any misconduct has occurred rests with the reviewer/instructor. Turnitin does not make a determination of misconduct, rather it provides data for the educators to make an informed decision based on their academic and institutional policies. 

Can I download the AI report like the Similarity report?

Yes. The AI detection report can be downloaded as a PDF via the ‘download’ button located in the right-hand corner of the report.

Can I view aggregated AI scores across submissions for my institution?

Yes, administrators can view aggregated AI scores for their institution. This feature is currently only available to admins and shows statistics at the institution/parent level. Please note that we are unable to differentiate between English and Spanish scores in this first iteration. We will look to add more granularity to these statistics as we continue to develop this capability further.

 

Scope of detection

Which AI writing models can Turnitin’s Spanish AI writing model detect?

This iteration of Turnitin’s Spanish AI writing detection capabilities has been trained to detect GPT-3.5 and  GPT-4. Our technology can also detect other AI writing tools that are based on these models such as ChatGPT. We plan to expand our detection capabilities to other models in the future.

Which model is Turnitin’s AI detection model based on?

Our model is based on a transformer-based classifier that was customized for the Spanish language. We undertook multiple rounds of carefully calibrated retraining, evaluation, and fine-tuning. What we must really emphasize is that the unique power of our model arises from the carefully curated data we've used to train the model, leveraging our 25+ years of expertise in authentic student writing, along with the technology developed by us to extract the maximum predictive power from the model trained on that data. In training our model, we focused on minimizing false positives while maximizing accuracy for the latest generation of LLMs ensuring that we help educators uphold academic integrity while protecting the interests of students.

Access & licensing

How can customers get access to AI writing detection? 

AI writing detection is only available to customers that license Turnitin Originality. If you are a Turnitin Similarity, Turnitin Feedback Studio (TFS) or Originality Check customer, please speak to your Turnitin account manager regarding access to AI writing detection.

Please note that for customers licensing TFS, the ‘OriginalityCheck’ listed within the products available when accessing your institutional account dashboard, refers to the component of your institution’s TFS license that allows for papers to be checked against our database, and generate Similarity Reports, and is distinct from Turnitin Originality.

iThenticate 2.0 customers can get access to this feature if they license AI writing capabilities as an add-on. Please speak to your account manager for details.

Is Turnitin’s AI writing detection a standalone solution or is it part of another product?

Turnitin’s AI writing detection capabilities are a separate feature of the Similarity Report and are available to customers when licensing Turnitin Originality in addition to their existing product, or the AI writing add-on, when using iThenticate 2.0.

Where can I find more information about this new solution?

You can find information about Turnitin’s AI writing detection capabilities.

Will Turnitin process my submission for AI writing detection if my institution does not use the feature?

No, we will only process submissions for AI writing detection if the institution has the feature enabled.

Can my institution opt out if we do not want Turnitin to process our submissions for AI writing detection?

Yes, if your institution does not want submissions to be processed for AI writing detection, they can opt out by disabling AI writing detection from their admin account settings page.

If I re-enable AI writing detection, will it automatically show me scores for submissions made before it was enabled?

No, we cannot retroactively process submissions. If you would like to process past submissions for AI writing, you will need to re-submit the document.

Can I request deletion of my institution’s data prior to disabling the feature?

Yes, customers can request a full deletion of their submissions; we cannot support partial data deletion requests to delete only the AI writing component of the submission data.