Li7ddlmZddlZddlZddlZddlmZddlZddl Z ddl m Z m Z ddl m Z ddlmZddlmZGdd e j$ZGd d e j$ZGd d Zy)) annotationsN)Iterable)Tensornn) InputExample)SentenceTransformer)cos_simc@eZdZdZdfd ZdddZeddZxZS) ContrastiveTensionLossa This loss expects only single sentences, without any labels. Positive and negative pairs are automatically created via random sampling, such that a positive pair consists of two identical sentences and a negative pair consists of two different sentences. An independent copy of the encoder model is created, which is used for encoding the first sentence of each pair. The original encoder model encodes the second sentence. The embeddings are compared and scored using the generated labels (1 if positive, 0 if negative) using the binary cross entropy objective. Generally, :class:`ContrastiveTensionLossInBatchNegatives` is recommended over this loss, as it gives a stronger training signal. Args: model: SentenceTransformer model References: * Semantic Re-Tuning with Contrastive Tension: https://openreview.net/pdf?id=Ov_sMNau-PF * `Unsupervised Learning > CT <../../../examples/sentence_transformer/unsupervised_learning/CT/README.html>`_ Inputs: +================================+=========+ | Texts | Labels | +================================+=========+ | (sentence_A, sentence_B) pairs | None | +================================+=========+ Relations: * :class:`ContrastiveTensionLossInBatchNegatives` uses in-batch negative sampling, which gives a stronger training signal than this loss. Example: Using a dataset with sentence pairs that sometimes are identical (positive pairs) and sometimes different (negative pairs): :: import random from datasets import Dataset from sentence_transformers import SentenceTransformer, losses from sentence_transformers.training_args import SentenceTransformerTrainingArguments from sentence_transformers.trainer import SentenceTransformerTrainer model = SentenceTransformer('all-MiniLM-L6-v2') # The dataset pairs must sometimes contain identical sentences (positive pairs) and sometimes different sentences (negative pairs). train_dataset = Dataset.from_dict({ "sentence_A": [ "It's nice weather outside today.", "He drove to work.", "It's so sunny.", ] "sentence_B": [ "It's nice weather outside today.", "He drove to work.", "It's so sunny.", ] }) train_loss = losses.ContrastiveTensionLoss(model=model) args = SentenceTransformerTrainingArguments( num_train_epochs=10, per_device_train_batch_size=32, eval_steps=0.1, logging_steps=0.01, learning_rate=5e-5, save_strategy="no", fp16=True, ) trainer = SentenceTransformerTrainer(model=model, args=args, train_dataset=train_dataset, loss=train_loss) trainer.train() With a dataset with single sentence pairs: :: import random from datasets import Dataset from sentence_transformers import SentenceTransformer, losses from sentence_transformers.training_args import SentenceTransformerTrainingArguments from sentence_transformers.trainer import SentenceTransformerTrainer model = SentenceTransformer('all-MiniLM-L6-v2') train_dataset = Dataset.from_dict({ "text1": [ "It's nice weather outside today.", "He drove to work.", "It's so sunny.", ] }) sentences = train_dataset['text1'] def to_ct_pairs(sample, pos_neg_ratio=8): pos_neg_ratio = 1 / pos_neg_ratio sample["text2"] = sample["text1"] if random.random() < pos_neg_ratio else random.choice(sentences) return sample pos_neg_ratio = 8 # 1 positive pair for 7 negative pairs train_dataset = train_dataset.map(to_ct_pairs, fn_kwargs={"pos_neg_ratio": pos_neg_ratio}) train_loss = losses.ContrastiveTensionLoss(model=model) args = SentenceTransformerTrainingArguments( num_train_epochs=10, per_device_train_batch_size=32, eval_steps=0.1, logging_steps=0.01, learning_rate=5e-5, save_strategy="no", fp16=True, ) trainer = SentenceTransformerTrainer(model=model, args=args, train_dataset=train_dataset, loss=train_loss) trainer.train() ct|||_tj||_t jd|_y)Nsum) reduction) super__init__model2copydeepcopymodel1rBCEWithLogitsLoss criterion)selfmodel __class__s y/var/www/html/lcp-python-backend/venv/lib/python3.12/site-packages/sentence_transformers/losses/ContrastiveTensionLoss.pyrzContrastiveTensionLoss.__init__s7  mmE* -->c t|\}}|j|d}|j|d}tj|dddf|dddddfj dj d}||d}|d} |j d} |j d} | | tjt|| j| | jD cgc]#\} } }}tj| | ||%c}}} } |j|j}n#|| k(jdj}|j||j!|}|Scc}}} } w)Nsentence_embedding input_idsattention_mask)devicedtype)dim)tuplerrtorchmatmulsqueezeget as_tensorzipboolequalr!r"allfloatrtype_as)rsentence_featureslabelssentence_features1sentence_features2reps_1reps_2 sim_scores input_ids1 input_ids2attention_mask1attention_mask2inputs1mask1inputs2mask2losss rforwardzContrastiveTensionLoss.forwards167H1I../01EF/01EF LL4&At*< = E Eb I Q QRT U  >+K8J+K8J0445EFO0445EFO*/J?B&(<(<(> OL`L`Lb?:GUGU GENGENC &,,$** % 277A7>DDF~~j&..*DE s,(E<cyNar @inproceedings{carlsson2021semantic, title={Semantic Re-tuning with Contrastive Tension}, author={Fredrik Carlsson and Amaru Cuba Gyllensten and Evangelia Gogoulou and Erik Ylip{"a}{"a} Hellqvist and Magnus Sahlgren}, booktitle={International Conference on Learning Representations}, year={2021}, url={https://openreview.net/forum?id=Ov_sMNau-PF} } rs rcitationzContrastiveTensionLoss.citationr)rrreturnNone)N)r1Iterable[dict[str, Tensor]]r2z Tensor | NonerHrrHstr) __name__ __module__ __qualname____doc__rrApropertyrF __classcell__rs@rr r s)kZ? D  rr c@eZdZdefdfd ZddZeddZxZS)&ContrastiveTensionLossInBatchNegativesg4@c*t|||_tj||_||_tj|_ tjtjgtj|z|_y)a This loss expects only single sentences, without any labels. Positive and negative pairs are automatically created via random sampling, such that a positive pair consists of two identical sentences and a negative pair consists of two different sentences. An independent copy of the encoder model is created, which is used for encoding the first sentence of each pair. The original encoder model encodes the second sentence. Unlike :class:`ContrastiveTensionLoss`, this loss uses the batch negative sampling strategy, i.e. the negative pairs are sampled from the batch. Using in-batch negative sampling gives a stronger training signal than the original :class:`ContrastiveTensionLoss`. The performance usually increases with increasing batch sizes. The training :class:`datasets.Dataset` must contain one text column. Args: model: SentenceTransformer model scale: Output of similarity function is multiplied by scale value similarity_fct: similarity function between sentence embeddings. By default, cos_sim. Can also be set to dot product (and then set scale to 1) References: - Semantic Re-Tuning with Contrastive Tension: https://openreview.net/pdf?id=Ov_sMNau-PF - `Unsupervised Learning > CT (In-Batch Negatives) <../../../examples/sentence_transformer/unsupervised_learning/CT_In-Batch_Negatives/README.html>`_ Relations: * :class:`ContrastiveTensionLoss` does not select negative pairs in-batch, resulting in a weaker training signal than this loss. Inputs: +------------------+--------+ | Texts | Labels | +==================+========+ | single sentences | none | +------------------+--------+ Example: :: from sentence_transformers import SentenceTransformer, losses from datasets import Dataset model = SentenceTransformer('all-MiniLM-L6-v2') train_dataset = Dataset.from_dict({ "sentence": [ "It's nice weather outside today.", "He drove to work.", "It's so sunny.", ] }) train_loss = losses.ContrastiveTensionLossInBatchNegatives(model=model) args = SentenceTransformerTrainingArguments( num_train_epochs=10, per_device_train_batch_size=32, eval_steps=0.1, logging_steps=0.01, learning_rate=5e-5, save_strategy="no", fp16=True, ) trainer = SentenceTransformerTrainer( model=model, args=args, train_dataset=train_dataset, loss=train_loss, ) trainer.train() N)rrrrrrsimilarity_fctrCrossEntropyLosscross_entropy_loss Parameterr&onesnplog logit_scale)rrscalerWrs rrz/ContrastiveTensionLossInBatchNegatives.__init__sgF  mmE* ,"$"5"5"7<< 2(FGrc|d}|j|d}|j|d}|j|||jj z}t j t|t j|j}|j|||j|j|zdz S)Nrr)r"r!) rrrWr^expr&arangelenlongr!rYt)rr1r2 embeddings_a embeddings_bscoress rrAz.ContrastiveTensionLossInBatchNegatives.forwards-a0{{#456JK {{#456JK $$\<@4CSCSCWCWCYYc&kFMMR''7$:Q:QRXRZRZR\^d:eeijjjrcyrCrDrEs rrFz/ContrastiveTensionLossInBatchNegatives.citationrGr)rrr_r/rHrI)r1rJr2rrHrrK) rMrNrOr rrArQrFrRrSs@rrUrUs+BFW^HHTk  rrUc eZdZddZdZdZy)ContrastiveTensionDataLoaderc||_||_||_d|_|j|jzdk7rt d|d|dy)Nrz@ContrastiveTensionDataLoader was loaded with a pos_neg_ratio of z and a batch size of z7. The batch size must be divisible by the pos_neg_ratio) sentences batch_size pos_neg_ratio collate_fn ValueError)rrnrorps rrz%ContrastiveTensionDataLoader.__init__so"$* ??T// /1 4RS`RaavxBwCCz{  5rc#"Ktj|jd}g}|dzt|jkr|j|}t||jzdkDr|dz }|j|}d}n|j|}d}|dz }|j t ||g|t||jk\r#|j|j|n|g}|dzt|jkryyw)Nrr#)textslabel) randomshufflernrdrpappendrrorq)r sentence_idxbatchs1s2rus r__iter__z%ContrastiveTensionDataLoader.__iter__'st~~& QT^^!44 -B5zD...2! ^^L1^^L1 A L LLRHEB C5zT__,040Kdooe,QVVQT^^!44s D D Dcrtjt|jd|jzz S)Nra)mathfloorrdrnrorEs r__len__z$ContrastiveTensionDataLoader.__len__=s(zz#dnn-T__1DEFFrN))rMrNrOrr}rrDrrrlrls ,Grrl) __future__rrrrvcollections.abcrnumpyr\r&rrsentence_transformers.readersr)sentence_transformers.SentenceTransformerrsentence_transformers.utilr Moduler rUrlrDrrrsX" $ 6I.`RYY`F^RYY^N#G#Gr