Causal Language Models in NLP

Causal Language Models in NLP

https://www.geeksforgeeks.org/nlp/causal-language-models-in-nlp/

Causal language models are a type of machine learning model that generates text by predicting the next word in a sequence based on the words that came before it. Unlike masked language models which predict missing words in a sentence by analyzing both preceding and succeeding words causal models operate in a unidirectional manner—processing text strictly from left to right or right to left.

These models are called "causal" because they rely on a causal relationship: each word depends only on the words that came before it not on any future words. This approach mimics how humans naturally process language as they read or speak.

The image explains how a Causal Language Model (CLM) predicts the next word using only previous words. The model takes "All," "the," "very," and "MASK" as input and predicts "best" for the masked word.

 

https://medium.com/@sajidc707/understanding-causal-and-masked-language-models-how-scaling-laws-impact-their-power-7768d8a86a68

Causal Language Model

Causal Language Modeling (CLM) is a type of language modeling where the model generates text in a sequential manner, predicting the next word based on the words it has already seen. It’s called “causal” because the model only uses past information to predict the future, just like how we naturally speak or write — by considering what we’ve said so far and predicting what comes next.

 

Association VS. Causal relationships

https://felixfan.github.io/association-VS-causal/

Association

• When two variables are related, we say that there is association between them.

When researchers find a correlation, which can also be called an association, what they are saying is that they found a relationship between two, or more, variables.

Causal

• One variable has a direct influence on the other, this is called a causal relationship.
• Causality can only be determined by reasoning about how the data were collected.
• The data values themselves contain no information that can help you to decide.

If two variables are causally related, it is possible to conclude that changes to the explanatory variable, X, will have a direct impact on Y.

If one variable causally affects the other, then adjusting the value of that variable will cause the other to change.

Obviously, it is much more difficult to prove causation than it is to prove an association.

Non-causal

In non-causal relationships, the relationship that is evident between the two variables is not completely the result of one variable directly affecting the other.

• Two variables can be related to each other without either variable directly affecting the values of the other.

If two variables are not causally related, it is impossible to tell whether changes to one variable, X, will result in changes to the other variable, Y.

Observational studies

• In observational studies, there is usually the potential for a lurking variable to underlie any observed relationship, so it is difficult to interpret relationships.

Data are collected in an observational study if we passively record (observe) values from each unit.

Experiments

• In a well designed experiment, there is little chance of lurking variables driving the observed relationships, so any relationship will be causal.

In an experiment, the researcher actively changes some characteristics of the units before the data are collected. The values of some variables are therefore under the control of the experimenter. In other words, the experimenter is able to choose each individual’s values for some variables.

References

http://www-ist.massey.ac.nz/dstirlin/CAST/CAST/Hcausal/causal_c1.html
http://www-ist.massey.ac.nz/dstirlin/CAST/CAST/Hcausal/causal_c2.html
http://www-ist.massey.ac.nz/dstirlin/CAST/CAST/Hcausal/causal_c3.html
http://www-ist.massey.ac.nz/dstirlin/CAST/CAST/Hcausal/causal_c4.html
http://www-ist.massey.ac.nz/dstirlin/CAST/CAST/Hcausal/causal_c5.html
Correlation, causation, and association - What does it all mean???